[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 08:57:09.817664 20382 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.231.190:40473
I20250114 08:57:09.819401 20382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 08:57:09.820583 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:09.838114 20388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:09.839674 20382 server_base.cc:1034] running on GCE node
W20250114 08:57:09.838802 20389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:09.841007 20391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:11.044174 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:11.044445 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:11.044716 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845031044691 us; error 0 us; skew 500 ppm
I20250114 08:57:11.045636 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:11.052654 20382 webserver.cc:458] Webserver started at http://127.19.231.190:43643/ using document root <none> and password file <none>
I20250114 08:57:11.053654 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:11.053866 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:11.054317 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:11.061070 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/master-0-root/instance:
uuid: "910dfd3b30cd4d1095fedb11d8f6fb46"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.069550 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250114 08:57:11.075083 20397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.076372 20382 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250114 08:57:11.076799 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/master-0-root
uuid: "910dfd3b30cd4d1095fedb11d8f6fb46"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.077180 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:11.132793 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:11.134387 20382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 08:57:11.134879 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:11.207387 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.190:40473
I20250114 08:57:11.207471 20448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.190:40473 every 8 connection(s)
I20250114 08:57:11.215163 20449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:11.240988 20449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46: Bootstrap starting.
I20250114 08:57:11.247047 20449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:11.249125 20449 log.cc:826] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46: Log is configured to *not* fsync() on all Append() calls
I20250114 08:57:11.254061 20449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46: No bootstrap required, opened a new log
I20250114 08:57:11.277527 20449 raft_consensus.cc:357] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" member_type: VOTER }
I20250114 08:57:11.278129 20449 raft_consensus.cc:383] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:11.278421 20449 raft_consensus.cc:738] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910dfd3b30cd4d1095fedb11d8f6fb46, State: Initialized, Role: FOLLOWER
I20250114 08:57:11.279292 20449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" member_type: VOTER }
I20250114 08:57:11.279830 20449 raft_consensus.cc:397] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 08:57:11.280110 20449 raft_consensus.cc:491] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 08:57:11.280450 20449 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:11.284674 20449 raft_consensus.cc:513] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" member_type: VOTER }
I20250114 08:57:11.285481 20449 leader_election.cc:304] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910dfd3b30cd4d1095fedb11d8f6fb46; no voters:
I20250114 08:57:11.287276 20449 leader_election.cc:290] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 08:57:11.287695 20452 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:11.290493 20452 raft_consensus.cc:695] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 1 LEADER]: Becoming Leader. State: Replica: 910dfd3b30cd4d1095fedb11d8f6fb46, State: Running, Role: LEADER
I20250114 08:57:11.291416 20452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" member_type: VOTER }
I20250114 08:57:11.292507 20449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 08:57:11.302352 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" member_type: VOTER } }
I20250114 08:57:11.303236 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [sys.catalog]: This master's current role is: LEADER
I20250114 08:57:11.303105 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910dfd3b30cd4d1095fedb11d8f6fb46. Latest consensus state: current_term: 1 leader_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910dfd3b30cd4d1095fedb11d8f6fb46" member_type: VOTER } }
I20250114 08:57:11.303884 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [sys.catalog]: This master's current role is: LEADER
I20250114 08:57:11.307597 20462 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 08:57:11.319067 20462 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 08:57:11.321053 20382 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 08:57:11.336300 20462 catalog_manager.cc:1348] Generated new cluster ID: c8f6ba9bbbad44c48081fa564006168f
I20250114 08:57:11.336666 20462 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 08:57:11.364356 20462 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 08:57:11.366674 20462 catalog_manager.cc:1505] Loading token signing keys...
I20250114 08:57:11.386394 20462 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46: Generated new TSK 0
I20250114 08:57:11.387351 20462 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 08:57:11.452138 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:11.461522 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:11.463239 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:11.466662 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:11.467845 20382 server_base.cc:1034] running on GCE node
I20250114 08:57:11.468814 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:11.469086 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:11.469287 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845031469270 us; error 0 us; skew 500 ppm
I20250114 08:57:11.470114 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:11.474503 20382 webserver.cc:458] Webserver started at http://127.19.231.129:39121/ using document root <none> and password file <none>
I20250114 08:57:11.475252 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:11.475497 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:11.475848 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:11.477389 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-0-root/instance:
uuid: "4d1a3fde922040f493631610e6158ecf"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.484441 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250114 08:57:11.489761 20479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.490628 20382 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250114 08:57:11.490918 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-0-root
uuid: "4d1a3fde922040f493631610e6158ecf"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.491218 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:11.514053 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:11.515532 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:11.518407 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:57:11.522591 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:57:11.522814 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.523106 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:57:11.523260 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.620427 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.129:35823
I20250114 08:57:11.620563 20541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.129:35823 every 8 connection(s)
I20250114 08:57:11.627893 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:11.644526 20546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:11.646111 20547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:11.650440 20549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:11.652817 20382 server_base.cc:1034] running on GCE node
I20250114 08:57:11.653863 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:11.654150 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:11.654337 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845031654321 us; error 0 us; skew 500 ppm
I20250114 08:57:11.655031 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:11.659009 20382 webserver.cc:458] Webserver started at http://127.19.231.130:45095/ using document root <none> and password file <none>
I20250114 08:57:11.659564 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:11.659766 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:11.660100 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:11.661641 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-1-root/instance:
uuid: "f6ccd6b41315457780eb97d4679f2b8b"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.663429 20542 heartbeater.cc:346] Connected to a master server at 127.19.231.190:40473
I20250114 08:57:11.663933 20542 heartbeater.cc:463] Registering TS with master...
I20250114 08:57:11.665132 20542 heartbeater.cc:510] Master 127.19.231.190:40473 requested a full tablet report, sending...
I20250114 08:57:11.668738 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250114 08:57:11.668769 20414 ts_manager.cc:194] Registered new tserver with Master: 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823)
I20250114 08:57:11.671912 20414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41764
I20250114 08:57:11.673357 20554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.674265 20382 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250114 08:57:11.674655 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-1-root
uuid: "f6ccd6b41315457780eb97d4679f2b8b"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.675030 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:11.694908 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:11.696327 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:11.698417 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:57:11.701364 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:57:11.701586 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.701817 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:57:11.701982 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.806164 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.130:33569
I20250114 08:57:11.806227 20616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.130:33569 every 8 connection(s)
I20250114 08:57:11.810873 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:11.822963 20620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:11.825151 20621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:11.825522 20617 heartbeater.cc:346] Connected to a master server at 127.19.231.190:40473
I20250114 08:57:11.825942 20617 heartbeater.cc:463] Registering TS with master...
I20250114 08:57:11.826812 20617 heartbeater.cc:510] Master 127.19.231.190:40473 requested a full tablet report, sending...
I20250114 08:57:11.829191 20414 ts_manager.cc:194] Registered new tserver with Master: f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:11.830941 20414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41768
W20250114 08:57:11.836376 20623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:11.837545 20382 server_base.cc:1034] running on GCE node
I20250114 08:57:11.838696 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:11.838969 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:11.839124 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845031839106 us; error 0 us; skew 500 ppm
I20250114 08:57:11.839656 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:11.842319 20382 webserver.cc:458] Webserver started at http://127.19.231.131:39761/ using document root <none> and password file <none>
I20250114 08:57:11.842808 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:11.842996 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:11.843235 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:11.844252 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-2-root/instance:
uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.849170 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250114 08:57:11.852779 20628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.853740 20382 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250114 08:57:11.854023 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-2-root
uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
format_stamp: "Formatted at 2025-01-14 08:57:11 on dist-test-slave-4wc2"
I20250114 08:57:11.854300 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736845029639699-20382-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:11.866194 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:11.867463 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:11.869635 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:57:11.872629 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:57:11.872841 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.873066 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:57:11.873227 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:11.966843 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.131:44967
I20250114 08:57:11.966965 20690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.131:44967 every 8 connection(s)
I20250114 08:57:11.985172 20691 heartbeater.cc:346] Connected to a master server at 127.19.231.190:40473
I20250114 08:57:11.985548 20691 heartbeater.cc:463] Registering TS with master...
I20250114 08:57:11.986357 20691 heartbeater.cc:510] Master 127.19.231.190:40473 requested a full tablet report, sending...
I20250114 08:57:11.988516 20414 ts_manager.cc:194] Registered new tserver with Master: 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:11.988788 20382 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016432568s
I20250114 08:57:11.990319 20414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41780
I20250114 08:57:11.991367 20382 test_util.cc:274] Using random seed: -1087805397
I20250114 08:57:12.054996 20414 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41794:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 08:57:12.060367 20414 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 08:57:12.169687 20582 tablet_service.cc:1467] Processing CreateTablet for tablet aba72626b63a421c9be0e746049b07a5 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250114 08:57:12.170408 20581 tablet_service.cc:1467] Processing CreateTablet for tablet 3b34b35aa1ef46f79000df66f3375c34 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250114 08:57:12.173280 20582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba72626b63a421c9be0e746049b07a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.175428 20581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b34b35aa1ef46f79000df66f3375c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.174737 20580 tablet_service.cc:1467] Processing CreateTablet for tablet f3cafaca6a7944139c6fab5add1daf89 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250114 08:57:12.182673 20580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3cafaca6a7944139c6fab5add1daf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.183189 20579 tablet_service.cc:1467] Processing CreateTablet for tablet a86050de79064e01a547176fae322bad (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250114 08:57:12.186967 20577 tablet_service.cc:1467] Processing CreateTablet for tablet b616441962804786a716c5861c31876b (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250114 08:57:12.191468 20576 tablet_service.cc:1467] Processing CreateTablet for tablet d8abd09fe96b445c93371b0eb694cee8 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250114 08:57:12.194402 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86050de79064e01a547176fae322bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.189069 20578 tablet_service.cc:1467] Processing CreateTablet for tablet 998e3a9cff6d4f52b09fe03d5a1fb31f (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250114 08:57:12.201104 20507 tablet_service.cc:1467] Processing CreateTablet for tablet aba72626b63a421c9be0e746049b07a5 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250114 08:57:12.204329 20577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b616441962804786a716c5861c31876b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.203737 20505 tablet_service.cc:1467] Processing CreateTablet for tablet f3cafaca6a7944139c6fab5add1daf89 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250114 08:57:12.208864 20507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba72626b63a421c9be0e746049b07a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.214673 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998e3a9cff6d4f52b09fe03d5a1fb31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.212874 20575 tablet_service.cc:1467] Processing CreateTablet for tablet a627ee6253c849afaf6ac1f6e82ab6e3 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250114 08:57:12.216259 20502 tablet_service.cc:1467] Processing CreateTablet for tablet b616441962804786a716c5861c31876b (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250114 08:57:12.210106 20506 tablet_service.cc:1467] Processing CreateTablet for tablet 3b34b35aa1ef46f79000df66f3375c34 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250114 08:57:12.225821 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3cafaca6a7944139c6fab5add1daf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.217171 20504 tablet_service.cc:1467] Processing CreateTablet for tablet a86050de79064e01a547176fae322bad (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250114 08:57:12.225782 20574 tablet_service.cc:1467] Processing CreateTablet for tablet fa033b09208d4422b4a55b35f4835ff1 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250114 08:57:12.227550 20576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8abd09fe96b445c93371b0eb694cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.231067 20501 tablet_service.cc:1467] Processing CreateTablet for tablet d8abd09fe96b445c93371b0eb694cee8 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250114 08:57:12.235158 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 998e3a9cff6d4f52b09fe03d5a1fb31f (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250114 08:57:12.236063 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b34b35aa1ef46f79000df66f3375c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.238238 20500 tablet_service.cc:1467] Processing CreateTablet for tablet a627ee6253c849afaf6ac1f6e82ab6e3 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250114 08:57:12.239001 20499 tablet_service.cc:1467] Processing CreateTablet for tablet fa033b09208d4422b4a55b35f4835ff1 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250114 08:57:12.241830 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a627ee6253c849afaf6ac1f6e82ab6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.244529 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa033b09208d4422b4a55b35f4835ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.246606 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a627ee6253c849afaf6ac1f6e82ab6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.245537 20574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa033b09208d4422b4a55b35f4835ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.262899 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8abd09fe96b445c93371b0eb694cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.272624 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998e3a9cff6d4f52b09fe03d5a1fb31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.279016 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86050de79064e01a547176fae322bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.304771 20502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b616441962804786a716c5861c31876b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:57:12.314069 20461 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:57:12.344094 20709 tablet_bootstrap.cc:492] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.352676 20656 tablet_service.cc:1467] Processing CreateTablet for tablet aba72626b63a421c9be0e746049b07a5 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250114 08:57:12.359972 20656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba72626b63a421c9be0e746049b07a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.362650 20653 tablet_service.cc:1467] Processing CreateTablet for tablet a86050de79064e01a547176fae322bad (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250114 08:57:12.369042 20654 tablet_service.cc:1467] Processing CreateTablet for tablet f3cafaca6a7944139c6fab5add1daf89 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250114 08:57:12.374339 20650 tablet_service.cc:1467] Processing CreateTablet for tablet d8abd09fe96b445c93371b0eb694cee8 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250114 08:57:12.380035 20651 tablet_service.cc:1467] Processing CreateTablet for tablet b616441962804786a716c5861c31876b (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250114 08:57:12.387428 20649 tablet_service.cc:1467] Processing CreateTablet for tablet a627ee6253c849afaf6ac1f6e82ab6e3 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250114 08:57:12.383417 20652 tablet_service.cc:1467] Processing CreateTablet for tablet 998e3a9cff6d4f52b09fe03d5a1fb31f (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250114 08:57:12.389645 20654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3cafaca6a7944139c6fab5add1daf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.381458 20655 tablet_service.cc:1467] Processing CreateTablet for tablet 3b34b35aa1ef46f79000df66f3375c34 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250114 08:57:12.390235 20648 tablet_service.cc:1467] Processing CreateTablet for tablet fa033b09208d4422b4a55b35f4835ff1 (DEFAULT_TABLE table=test-workload [id=a534d5f532354efabfb2aa50ed6bb401]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250114 08:57:12.405337 20653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86050de79064e01a547176fae322bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.408013 20655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b34b35aa1ef46f79000df66f3375c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.415148 20649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a627ee6253c849afaf6ac1f6e82ab6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.433809 20648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa033b09208d4422b4a55b35f4835ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.461807 20651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b616441962804786a716c5861c31876b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.466571 20652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998e3a9cff6d4f52b09fe03d5a1fb31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.486083 20650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8abd09fe96b445c93371b0eb694cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:12.499588 20709 tablet_bootstrap.cc:654] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.538846 20710 tablet_bootstrap.cc:492] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:12.553611 20709 tablet_bootstrap.cc:492] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.554275 20709 ts_tablet_manager.cc:1397] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.211s user 0.021s sys 0.008s
I20250114 08:57:12.557626 20712 tablet_bootstrap.cc:492] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:12.566330 20710 tablet_bootstrap.cc:654] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.571573 20709 raft_consensus.cc:357] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.573311 20709 raft_consensus.cc:383] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.573670 20709 raft_consensus.cc:738] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.574528 20709 consensus_queue.cc:260] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.583287 20712 tablet_bootstrap.cc:654] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.586081 20709 ts_tablet_manager.cc:1428] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20250114 08:57:12.587034 20617 heartbeater.cc:502] Master 127.19.231.190:40473 was elected leader, sending a full tablet report...
I20250114 08:57:12.587321 20709 tablet_bootstrap.cc:492] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.590806 20712 tablet_bootstrap.cc:492] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:12.591262 20712 ts_tablet_manager.cc:1397] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.005s
I20250114 08:57:12.594103 20709 tablet_bootstrap.cc:654] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.594285 20712 raft_consensus.cc:357] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.595063 20712 raft_consensus.cc:383] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.595340 20712 raft_consensus.cc:738] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.596117 20712 consensus_queue.cc:260] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.597932 20710 tablet_bootstrap.cc:492] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:12.598471 20710 ts_tablet_manager.cc:1397] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.007s
I20250114 08:57:12.601938 20710 raft_consensus.cc:357] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.602980 20710 raft_consensus.cc:383] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.603345 20710 raft_consensus.cc:738] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.604275 20710 consensus_queue.cc:260] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.607686 20691 heartbeater.cc:502] Master 127.19.231.190:40473 was elected leader, sending a full tablet report...
I20250114 08:57:12.608378 20712 ts_tablet_manager.cc:1428] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
I20250114 08:57:12.609349 20712 tablet_bootstrap.cc:492] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:12.622926 20542 heartbeater.cc:502] Master 127.19.231.190:40473 was elected leader, sending a full tablet report...
I20250114 08:57:12.627683 20712 tablet_bootstrap.cc:654] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.628780 20709 tablet_bootstrap.cc:492] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.630417 20709 ts_tablet_manager.cc:1397] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.008s
W20250114 08:57:12.630921 20543 tablet.cc:2367] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 08:57:12.629120 20710 ts_tablet_manager.cc:1428] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.030s user 0.015s sys 0.002s
I20250114 08:57:12.632153 20710 tablet_bootstrap.cc:492] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:12.634271 20713 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:12.634764 20713 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.637691 20713 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:12.643415 20709 raft_consensus.cc:357] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.644367 20709 raft_consensus.cc:383] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.647783 20709 raft_consensus.cc:738] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.648622 20709 consensus_queue.cc:260] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.651293 20709 ts_tablet_manager.cc:1428] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250114 08:57:12.647485 20710 tablet_bootstrap.cc:654] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.655591 20709 tablet_bootstrap.cc:492] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.666088 20709 tablet_bootstrap.cc:654] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.675652 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
W20250114 08:57:12.677891 20557 leader_election.cc:343] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.680163 20709 tablet_bootstrap.cc:492] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.682415 20709 ts_tablet_manager.cc:1397] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20250114 08:57:12.685341 20709 raft_consensus.cc:357] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.686180 20709 raft_consensus.cc:383] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.686491 20709 raft_consensus.cc:738] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.687239 20709 consensus_queue.cc:260] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.696122 20709 ts_tablet_manager.cc:1428] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250114 08:57:12.699009 20709 tablet_bootstrap.cc:492] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.710469 20709 tablet_bootstrap.cc:654] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.713789 20718 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:12.714349 20718 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.716979 20718 leader_election.cc:290] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:12.721866 20709 tablet_bootstrap.cc:492] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.722386 20709 ts_tablet_manager.cc:1397] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20250114 08:57:12.725301 20709 raft_consensus.cc:357] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.726047 20709 raft_consensus.cc:383] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.726317 20709 raft_consensus.cc:738] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.726953 20709 consensus_queue.cc:260] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.737135 20710 tablet_bootstrap.cc:492] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:12.737680 20710 ts_tablet_manager.cc:1397] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.106s user 0.020s sys 0.024s
I20250114 08:57:12.740952 20710 raft_consensus.cc:357] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.745213 20713 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:12.745718 20713 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.747783 20713 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:12.745071 20709 ts_tablet_manager.cc:1428] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.022s user 0.009s sys 0.000s
I20250114 08:57:12.748589 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:12.749200 20709 tablet_bootstrap.cc:492] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
W20250114 08:57:12.750178 20557 leader_election.cc:343] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.741735 20710 raft_consensus.cc:383] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.752871 20710 raft_consensus.cc:738] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.753579 20710 consensus_queue.cc:260] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.756314 20710 ts_tablet_manager.cc:1428] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.018s user 0.002s sys 0.006s
I20250114 08:57:12.757277 20710 tablet_bootstrap.cc:492] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:12.766599 20712 tablet_bootstrap.cc:492] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:12.768674 20712 ts_tablet_manager.cc:1397] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.160s user 0.045s sys 0.055s
I20250114 08:57:12.768779 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:12.771517 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
W20250114 08:57:12.774044 20558 leader_election.cc:343] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.774524 20558 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:12.777503 20709 tablet_bootstrap.cc:654] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.777508 20713 raft_consensus.cc:2743] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 08:57:12.778080 20558 leader_election.cc:343] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.779357 20558 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:12.778051 20712 raft_consensus.cc:357] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.780658 20712 raft_consensus.cc:383] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.782078 20713 raft_consensus.cc:2743] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:12.781061 20712 raft_consensus.cc:738] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.783250 20712 consensus_queue.cc:260] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.786516 20712 ts_tablet_manager.cc:1428] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.017s user 0.005s sys 0.002s
I20250114 08:57:12.788626 20712 tablet_bootstrap.cc:492] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:12.791375 20710 tablet_bootstrap.cc:654] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.806744 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:12.810546 20709 tablet_bootstrap.cc:492] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
W20250114 08:57:12.810802 20483 leader_election.cc:343] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.811023 20712 tablet_bootstrap.cc:654] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.811072 20709 ts_tablet_manager.cc:1397] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.007s
I20250114 08:57:12.817683 20709 raft_consensus.cc:357] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.818491 20709 raft_consensus.cc:383] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.818830 20709 raft_consensus.cc:738] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.820489 20709 consensus_queue.cc:260] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.825870 20718 raft_consensus.cc:491] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:12.827934 20718 raft_consensus.cc:513] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.832508 20709 ts_tablet_manager.cc:1428] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.021s user 0.010s sys 0.000s
I20250114 08:57:12.833953 20709 tablet_bootstrap.cc:492] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.841902 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:12.843052 20592 raft_consensus.cc:2463] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 0.
I20250114 08:57:12.844585 20483 leader_election.cc:304] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:12.854127 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:12.854477 20731 raft_consensus.cc:2798] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:12.854966 20731 raft_consensus.cc:491] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:12.855311 20709 tablet_bootstrap.cc:654] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
W20250114 08:57:12.855758 20483 leader_election.cc:343] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.856267 20710 tablet_bootstrap.cc:492] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:12.856220 20483 leader_election.cc:304] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf; no voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:12.856700 20718 leader_election.cc:290] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:12.856837 20710 ts_tablet_manager.cc:1397] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.100s user 0.030s sys 0.012s
I20250114 08:57:12.857213 20718 raft_consensus.cc:2743] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:12.855340 20731 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:12.861773 20712 tablet_bootstrap.cc:492] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:12.862704 20712 ts_tablet_manager.cc:1397] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.007s
I20250114 08:57:12.866430 20712 raft_consensus.cc:357] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.867487 20712 raft_consensus.cc:383] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.867880 20712 raft_consensus.cc:738] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.870015 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
W20250114 08:57:12.871547 20483 leader_election.cc:343] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.875389 20709 tablet_bootstrap.cc:492] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.876011 20709 ts_tablet_manager.cc:1397] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.008s
I20250114 08:57:12.875999 20712 consensus_queue.cc:260] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.878255 20710 raft_consensus.cc:357] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.879082 20710 raft_consensus.cc:383] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.879418 20710 raft_consensus.cc:738] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.879703 20709 raft_consensus.cc:357] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.880185 20710 consensus_queue.cc:260] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.880940 20709 raft_consensus.cc:383] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.881388 20709 raft_consensus.cc:738] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.882297 20709 consensus_queue.cc:260] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.885128 20710 ts_tablet_manager.cc:1428] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20250114 08:57:12.886092 20710 tablet_bootstrap.cc:492] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:12.886157 20712 ts_tablet_manager.cc:1428] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20250114 08:57:12.887423 20709 ts_tablet_manager.cc:1428] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250114 08:57:12.888340 20709 tablet_bootstrap.cc:492] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.875170 20731 raft_consensus.cc:513] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.893311 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:12.893934 20592 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:12.899350 20709 tablet_bootstrap.cc:654] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.901612 20592 raft_consensus.cc:2463] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 1.
I20250114 08:57:12.902802 20483 leader_election.cc:304] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:12.903609 20712 tablet_bootstrap.cc:492] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:12.903970 20732 raft_consensus.cc:2798] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:12.904567 20732 raft_consensus.cc:695] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:12.905421 20732 consensus_queue.cc:237] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.911585 20712 tablet_bootstrap.cc:654] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.912593 20731 leader_election.cc:290] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:12.912824 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
W20250114 08:57:12.914197 20483 leader_election.cc:343] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:12.912287 20709 tablet_bootstrap.cc:492] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.915087 20709 ts_tablet_manager.cc:1397] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.008s
I20250114 08:57:12.928076 20710 tablet_bootstrap.cc:654] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.927183 20413 catalog_manager.cc:5526] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 1 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:12.933135 20709 raft_consensus.cc:357] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.933939 20709 raft_consensus.cc:383] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.934263 20709 raft_consensus.cc:738] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.934970 20709 consensus_queue.cc:260] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.937845 20709 ts_tablet_manager.cc:1428] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250114 08:57:12.939657 20709 tablet_bootstrap.cc:492] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.947968 20709 tablet_bootstrap.cc:654] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.951889 20712 tablet_bootstrap.cc:492] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:12.952371 20712 ts_tablet_manager.cc:1397] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.011s
I20250114 08:57:12.954486 20710 tablet_bootstrap.cc:492] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:12.955094 20710 ts_tablet_manager.cc:1397] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.006s
I20250114 08:57:12.954985 20712 raft_consensus.cc:357] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.956132 20717 raft_consensus.cc:491] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:12.955806 20712 raft_consensus.cc:383] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.958397 20712 raft_consensus.cc:738] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.959198 20712 consensus_queue.cc:260] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.956784 20717 raft_consensus.cc:513] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.961162 20710 raft_consensus.cc:357] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.962103 20710 raft_consensus.cc:383] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.962699 20717 leader_election.cc:290] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:12.962458 20710 raft_consensus.cc:738] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.964841 20710 consensus_queue.cc:260] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.970320 20710 ts_tablet_manager.cc:1428] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20250114 08:57:12.971637 20710 tablet_bootstrap.cc:492] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:12.979701 20710 tablet_bootstrap.cc:654] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:12.982717 20709 tablet_bootstrap.cc:492] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:12.983326 20709 ts_tablet_manager.cc:1397] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.005s
I20250114 08:57:12.986485 20709 raft_consensus.cc:357] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.987409 20709 raft_consensus.cc:383] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:12.987747 20709 raft_consensus.cc:738] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:12.988591 20709 consensus_queue.cc:260] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.991616 20709 ts_tablet_manager.cc:1428] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250114 08:57:12.992751 20713 raft_consensus.cc:491] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:12.993211 20713 raft_consensus.cc:513] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:12.995646 20713 leader_election.cc:290] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:12.997282 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:12.997934 20517 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 0.
I20250114 08:57:12.999125 20709 tablet_bootstrap.cc:492] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b: Bootstrap starting.
I20250114 08:57:12.999523 20557 leader_election.cc:304] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.000373 20713 raft_consensus.cc:2798] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.000766 20713 raft_consensus.cc:491] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.001070 20713 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.001667 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.002337 20666 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 0.
I20250114 08:57:13.009451 20713 raft_consensus.cc:513] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.017241 20713 leader_election.cc:290] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.017911 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:13.018553 20517 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.020736 20709 tablet_bootstrap.cc:654] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.021768 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.022482 20666 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.028302 20712 ts_tablet_manager.cc:1428] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.076s user 0.005s sys 0.006s
I20250114 08:57:13.030573 20712 tablet_bootstrap.cc:492] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:13.033951 20666 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 1.
I20250114 08:57:13.036604 20558 leader_election.cc:304] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.037775 20713 raft_consensus.cc:2798] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.044139 20713 raft_consensus.cc:695] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:13.050139 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa033b09208d4422b4a55b35f4835ff1" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:13.050187 20517 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 1.
I20250114 08:57:13.053951 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa033b09208d4422b4a55b35f4835ff1" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.054629 20592 raft_consensus.cc:2463] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 0.
I20250114 08:57:13.054931 20710 tablet_bootstrap.cc:492] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:13.056114 20632 leader_election.cc:304] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.055522 20710 ts_tablet_manager.cc:1397] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.084s user 0.016s sys 0.017s
I20250114 08:57:13.058498 20743 raft_consensus.cc:2798] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250114 08:57:13.059307 20631 leader_election.cc:343] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.060985 20743 raft_consensus.cc:491] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.061487 20743 raft_consensus.cc:3054] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.062786 20717 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.053057 20713 consensus_queue.cc:237] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.065136 20717 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.067687 20717 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.069494 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.067981 20710 raft_consensus.cc:357] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.070969 20710 raft_consensus.cc:383] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.070927 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:13.071405 20710 raft_consensus.cc:738] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.071906 20712 tablet_bootstrap.cc:654] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.070304 20592 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 0.
I20250114 08:57:13.073083 20632 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.068656 20746 raft_consensus.cc:491] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.074708 20717 raft_consensus.cc:2798] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250114 08:57:13.074698 20631 leader_election.cc:343] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.075873 20709 tablet_bootstrap.cc:492] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b: No bootstrap required, opened a new log
I20250114 08:57:13.076241 20709 ts_tablet_manager.cc:1397] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.019s
I20250114 08:57:13.075270 20717 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.076779 20717 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.074810 20746 raft_consensus.cc:513] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.081504 20709 raft_consensus.cc:357] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.082019 20709 raft_consensus.cc:383] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.082228 20709 raft_consensus.cc:738] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.082798 20709 consensus_queue.cc:260] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.072384 20710 consensus_queue.cc:260] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.089037 20710 ts_tablet_manager.cc:1428] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20250114 08:57:13.090655 20710 tablet_bootstrap.cc:492] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:13.087240 20743 raft_consensus.cc:513] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.100772 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:13.101307 20743 leader_election.cc:290] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.102366 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa033b09208d4422b4a55b35f4835ff1" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.105424 20592 raft_consensus.cc:3054] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:57:13.104187 20557 leader_election.cc:343] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.122224 20732 raft_consensus.cc:491] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.123158 20732 raft_consensus.cc:513] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.126448 20732 leader_election.cc:290] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.125329 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa033b09208d4422b4a55b35f4835ff1" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:13.126677 20717 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.129621 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b616441962804786a716c5861c31876b" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.129946 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b616441962804786a716c5861c31876b" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.139802 20591 raft_consensus.cc:2463] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 0.
W20250114 08:57:13.142736 20483 leader_election.cc:343] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.132663 20746 leader_election.cc:290] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.143821 20746 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.144001 20483 leader_election.cc:304] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters: 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:13.129709 20592 raft_consensus.cc:2463] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 1.
W20250114 08:57:13.130043 20631 leader_election.cc:343] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.145485 20732 raft_consensus.cc:2798] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.146283 20732 raft_consensus.cc:491] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.147245 20732 raft_consensus.cc:3054] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.147858 20632 leader_election.cc:304] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.144286 20746 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.149727 20743 raft_consensus.cc:2798] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.150367 20709 ts_tablet_manager.cc:1428] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b: Time spent starting tablet: real 0.074s user 0.027s sys 0.011s
I20250114 08:57:13.152518 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf"
W20250114 08:57:13.154328 20631 leader_election.cc:343] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.158463 20746 leader_election.cc:290] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.158880 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:13.159842 20517 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 0.
I20250114 08:57:13.160068 20710 tablet_bootstrap.cc:654] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.161661 20557 leader_election.cc:304] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.167099 20712 tablet_bootstrap.cc:492] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:13.168390 20712 ts_tablet_manager.cc:1397] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.138s user 0.025s sys 0.017s
I20250114 08:57:13.167609 20746 raft_consensus.cc:2798] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.168994 20746 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.169531 20746 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.170060 20732 raft_consensus.cc:513] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.174193 20743 raft_consensus.cc:695] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.174502 20717 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.177578 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.178210 20592 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.176638 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.181885 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b616441962804786a716c5861c31876b" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.182469 20591 raft_consensus.cc:3054] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.183666 20712 raft_consensus.cc:357] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.184696 20712 raft_consensus.cc:383] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.185070 20712 raft_consensus.cc:738] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.183279 20752 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.187410 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b616441962804786a716c5861c31876b" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.188045 20732 leader_election.cc:290] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.189399 20752 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
W20250114 08:57:13.189947 20483 leader_election.cc:343] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.189716 20712 consensus_queue.cc:260] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.192673 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.183550 20743 consensus_queue.cc:237] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.194054 20412 catalog_manager.cc:5526] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 1 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.197676 20712 ts_tablet_manager.cc:1428] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250114 08:57:13.198670 20712 tablet_bootstrap.cc:492] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:13.202461 20591 raft_consensus.cc:2463] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 1.
I20250114 08:57:13.203759 20710 tablet_bootstrap.cc:492] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:13.203809 20483 leader_election.cc:304] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters: 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:13.204327 20710 ts_tablet_manager.cc:1397] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.017s
I20250114 08:57:13.205169 20732 raft_consensus.cc:2798] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.205667 20732 raft_consensus.cc:695] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:13.206490 20732 consensus_queue.cc:237] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
W20250114 08:57:13.195717 20558 leader_election.cc:343] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.207166 20710 raft_consensus.cc:357] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.207605 20558 leader_election.cc:304] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:13.207955 20710 raft_consensus.cc:383] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.201622 20746 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.208731 20751 raft_consensus.cc:2743] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 08:57:13.209015 20558 leader_election.cc:343] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.209853 20592 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 1.
I20250114 08:57:13.210841 20746 leader_election.cc:290] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.208261 20710 raft_consensus.cc:738] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.211220 20632 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.212867 20731 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.213450 20717 raft_consensus.cc:2798] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.213531 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:13.213316 20731 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.214174 20517 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.216620 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.218421 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
W20250114 08:57:13.218212 20558 leader_election.cc:343] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.219841 20516 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 0.
I20250114 08:57:13.215243 20710 consensus_queue.cc:260] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.227766 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.228583 20592 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 0.
I20250114 08:57:13.230032 20710 ts_tablet_manager.cc:1428] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250114 08:57:13.230592 20483 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.231009 20710 tablet_bootstrap.cc:492] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:13.231420 20718 raft_consensus.cc:2798] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.231804 20718 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.232137 20718 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.238099 20631 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:13.233515 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.241111 20591 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 0.
I20250114 08:57:13.241405 20743 raft_consensus.cc:2798] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.241776 20743 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.242128 20743 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.263554 20731 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.232540 20752 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.268061 20712 tablet_bootstrap.cc:654] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.269876 20743 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.271009 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.271885 20666 raft_consensus.cc:2388] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d1a3fde922040f493631610e6158ecf in current term 1: Already voted for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in this term.
I20250114 08:57:13.272605 20743 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.273663 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:13.275537 20718 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.277988 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.278795 20591 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.279232 20413 catalog_manager.cc:5526] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 1 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.283205 20717 raft_consensus.cc:695] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.277071 20516 raft_consensus.cc:2388] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 95fdaef96b2b44139ac68afc82bbf1d4 in current term 1: Already voted for candidate 4d1a3fde922040f493631610e6158ecf in this term.
I20250114 08:57:13.284732 20717 consensus_queue.cc:237] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.289255 20517 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 1.
I20250114 08:57:13.293203 20718 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.293509 20557 leader_election.cc:304] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters: 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:13.294603 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.294771 20746 raft_consensus.cc:2798] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.288297 20710 tablet_bootstrap.cc:654] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.295688 20666 raft_consensus.cc:2388] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d1a3fde922040f493631610e6158ecf in current term 1: Already voted for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in this term.
I20250114 08:57:13.295528 20746 raft_consensus.cc:695] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:13.297983 20746 consensus_queue.cc:237] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.296653 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.300794 20483 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf; no voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.302906 20718 raft_consensus.cc:2743] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 08:57:13.305564 20712 tablet_bootstrap.cc:492] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:13.306264 20712 ts_tablet_manager.cc:1397] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.108s user 0.020s sys 0.000s
I20250114 08:57:13.305634 20414 catalog_manager.cc:5526] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:13.309823 20712 raft_consensus.cc:357] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.310989 20712 raft_consensus.cc:383] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.311331 20712 raft_consensus.cc:738] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.315822 20710 tablet_bootstrap.cc:492] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:13.316330 20710 ts_tablet_manager.cc:1397] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.086s user 0.019s sys 0.006s
W20250114 08:57:13.316571 20461 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:57:13.319036 20710 raft_consensus.cc:357] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.319823 20710 raft_consensus.cc:383] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.320122 20710 raft_consensus.cc:738] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.321408 20710 consensus_queue.cc:260] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.326925 20710 ts_tablet_manager.cc:1428] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20250114 08:57:13.327955 20710 tablet_bootstrap.cc:492] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:13.328612 20712 consensus_queue.cc:260] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.332330 20756 raft_consensus.cc:491] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.332857 20756 raft_consensus.cc:513] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.335059 20756 leader_election.cc:290] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.336997 20712 ts_tablet_manager.cc:1428] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.030s user 0.010s sys 0.000s
I20250114 08:57:13.336851 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.337875 20592 raft_consensus.cc:2463] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 0.
I20250114 08:57:13.339327 20632 leader_election.cc:304] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.338093 20712 tablet_bootstrap.cc:492] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:13.340358 20756 raft_consensus.cc:2798] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.340834 20756 raft_consensus.cc:491] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.341195 20756 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.341034 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
W20250114 08:57:13.342882 20631 leader_election.cc:343] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.347929 20591 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 1.
I20250114 08:57:13.355656 20717 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.355892 20632 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.351665 20756 raft_consensus.cc:513] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.356352 20717 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.357687 20752 raft_consensus.cc:2798] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.358179 20752 raft_consensus.cc:695] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.359555 20717 leader_election.cc:290] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.362562 20710 tablet_bootstrap.cc:654] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.364416 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.364862 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:13.365700 20517 raft_consensus.cc:2388] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95fdaef96b2b44139ac68afc82bbf1d4 in current term 1: Already voted for candidate f6ccd6b41315457780eb97d4679f2b8b in this term.
I20250114 08:57:13.366626 20631 leader_election.cc:304] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4; no voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.367445 20743 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.367012 20412 catalog_manager.cc:5526] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:13.363214 20752 consensus_queue.cc:237] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.375205 20743 raft_consensus.cc:2743] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:13.363601 20413 catalog_manager.cc:5526] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 1 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.373787 20712 tablet_bootstrap.cc:654] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.376441 20756 leader_election.cc:290] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.379025 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf"
W20250114 08:57:13.380323 20631 leader_election.cc:343] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:13.378386 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.386977 20591 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.403685 20412 catalog_manager.cc:5526] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:13.407629 20718 consensus_queue.cc:1035] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:13.412814 20591 raft_consensus.cc:2463] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 1.
I20250114 08:57:13.414613 20632 leader_election.cc:304] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters: 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.415647 20752 raft_consensus.cc:2798] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.416152 20752 raft_consensus.cc:695] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.417133 20752 consensus_queue.cc:237] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.418277 20710 tablet_bootstrap.cc:492] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:13.418877 20710 ts_tablet_manager.cc:1397] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.019s
I20250114 08:57:13.423454 20710 raft_consensus.cc:357] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.424326 20712 tablet_bootstrap.cc:492] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:13.424531 20710 raft_consensus.cc:383] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.425060 20710 raft_consensus.cc:738] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.425134 20712 ts_tablet_manager.cc:1397] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.033s
I20250114 08:57:13.425899 20710 consensus_queue.cc:260] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.428879 20712 raft_consensus.cc:357] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.429678 20712 raft_consensus.cc:383] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.430150 20712 raft_consensus.cc:738] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.430411 20710 ts_tablet_manager.cc:1428] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250114 08:57:13.430915 20712 consensus_queue.cc:260] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.432786 20710 tablet_bootstrap.cc:492] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: Bootstrap starting.
I20250114 08:57:13.433827 20712 ts_tablet_manager.cc:1428] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20250114 08:57:13.434849 20712 tablet_bootstrap.cc:492] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:13.438501 20710 tablet_bootstrap.cc:654] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.441716 20712 tablet_bootstrap.cc:654] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.445966 20411 catalog_manager.cc:5526] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:13.449939 20710 tablet_bootstrap.cc:492] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: No bootstrap required, opened a new log
I20250114 08:57:13.450464 20710 ts_tablet_manager.cc:1397] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20250114 08:57:13.452175 20712 tablet_bootstrap.cc:492] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:13.452672 20712 ts_tablet_manager.cc:1397] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250114 08:57:13.453537 20710 raft_consensus.cc:357] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.454260 20710 raft_consensus.cc:383] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.454526 20710 raft_consensus.cc:738] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.455394 20712 raft_consensus.cc:357] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.455303 20710 consensus_queue.cc:260] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.456107 20712 raft_consensus.cc:383] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.456394 20712 raft_consensus.cc:738] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.457059 20712 consensus_queue.cc:260] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.457985 20710 ts_tablet_manager.cc:1428] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250114 08:57:13.459932 20712 ts_tablet_manager.cc:1428] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 08:57:13.460855 20712 tablet_bootstrap.cc:492] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4: Bootstrap starting.
I20250114 08:57:13.466864 20712 tablet_bootstrap.cc:654] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:13.476823 20712 tablet_bootstrap.cc:492] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4: No bootstrap required, opened a new log
I20250114 08:57:13.477327 20712 ts_tablet_manager.cc:1397] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
W20250114 08:57:13.479761 20483 consensus_peers.cc:487] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf -> Peer 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967): Couldn't send request to peer 95fdaef96b2b44139ac68afc82bbf1d4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:57:13.479846 20712 raft_consensus.cc:357] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.480666 20712 raft_consensus.cc:383] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:13.480963 20712 raft_consensus.cc:738] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Initialized, Role: FOLLOWER
I20250114 08:57:13.481657 20712 consensus_queue.cc:260] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.483640 20712 ts_tablet_manager.cc:1428] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 08:57:13.493453 20731 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.493965 20731 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.495980 20731 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.497104 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.497283 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.497797 20592 raft_consensus.cc:2388] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d1a3fde922040f493631610e6158ecf in current term 1: Already voted for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in this term.
I20250114 08:57:13.498915 20483 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf; no voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.499650 20731 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.507284 20731 raft_consensus.cc:2743] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:13.549125 20746 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:13.559777 20732 raft_consensus.cc:491] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.560231 20713 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:13.560292 20732 raft_consensus.cc:513] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.563750 20732 leader_election.cc:290] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.566653 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.567435 20665 raft_consensus.cc:2463] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 0.
I20250114 08:57:13.569110 20483 leader_election.cc:304] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:13.569887 20732 raft_consensus.cc:2798] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:13.570268 20732 raft_consensus.cc:491] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:13.570634 20732 raft_consensus.cc:3054] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.566617 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.571806 20592 raft_consensus.cc:2463] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 0.
I20250114 08:57:13.579020 20732 raft_consensus.cc:513] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.581765 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.582376 20592 raft_consensus.cc:3054] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.582424 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.583032 20665 raft_consensus.cc:3054] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.586076 20732 leader_election.cc:290] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.589637 20665 raft_consensus.cc:2463] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 1.
I20250114 08:57:13.590719 20592 raft_consensus.cc:2463] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 1.
I20250114 08:57:13.590843 20483 leader_election.cc:304] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:13.591838 20732 raft_consensus.cc:2798] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:13.592931 20732 raft_consensus.cc:695] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:13.593770 20732 consensus_queue.cc:237] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.601588 20412 catalog_manager.cc:5526] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 1 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.625885 20732 raft_consensus.cc:491] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.626391 20732 raft_consensus.cc:513] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.627969 20732 leader_election.cc:290] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.629103 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa033b09208d4422b4a55b35f4835ff1" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.629033 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa033b09208d4422b4a55b35f4835ff1" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.629997 20592 raft_consensus.cc:2388] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d1a3fde922040f493631610e6158ecf in current term 1: Already voted for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in this term.
I20250114 08:57:13.630956 20483 leader_election.cc:304] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf; no voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.631788 20732 raft_consensus.cc:3054] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.636009 20732 raft_consensus.cc:2743] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:13.670398 20752 consensus_queue.cc:1035] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.683115 20756 consensus_queue.cc:1035] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.713272 20665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
mode: GRACEFUL
new_leader_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:57918
I20250114 08:57:13.714085 20665 raft_consensus.cc:604] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Received request to transfer leadership to TS 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.722168 20665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
mode: GRACEFUL
new_leader_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:57918
I20250114 08:57:13.722818 20665 raft_consensus.cc:604] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Received request to transfer leadership to TS 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.732861 20756 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.738194 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a86050de79064e01a547176fae322bad"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:37956
I20250114 08:57:13.739022 20592 raft_consensus.cc:604] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Received request to transfer leadership to TS 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:13.743710 20731 consensus_queue.cc:1035] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.754477 20717 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.769412 20717 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.786908 20665 raft_consensus.cc:3054] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.788750 20756 raft_consensus.cc:988] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: : Instructing follower 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:13.789199 20756 raft_consensus.cc:1076] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Signalling peer 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:13.796242 20743 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.798173 20517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:41682
I20250114 08:57:13.798794 20517 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:13.799163 20517 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.802341 20718 consensus_queue.cc:1035] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.807526 20752 raft_consensus.cc:491] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.808002 20752 raft_consensus.cc:513] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.810405 20752 leader_election.cc:290] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:13.813071 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.814021 20515 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:41692
I20250114 08:57:13.814795 20515 raft_consensus.cc:604] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.816277 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1a3fde922040f493631610e6158ecf" is_pre_election: true
I20250114 08:57:13.820816 20631 leader_election.cc:304] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4; no voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.821931 20717 raft_consensus.cc:2743] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:13.827812 20517 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.831645 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.832684 20517 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 2 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.833292 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.833897 20666 raft_consensus.cc:3049] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:13.834275 20666 raft_consensus.cc:738] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.832267 20592 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.835824 20666 consensus_queue.cc:260] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.836552 20517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b616441962804786a716c5861c31876b"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:41692
I20250114 08:57:13.837167 20517 raft_consensus.cc:604] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.837311 20666 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.838222 20718 raft_consensus.cc:491] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:13.838660 20718 raft_consensus.cc:513] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.840579 20718 leader_election.cc:290] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.842679 20664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" is_pre_election: true
I20250114 08:57:13.844887 20717 consensus_queue.cc:1035] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.845147 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b" is_pre_election: true
I20250114 08:57:13.845885 20591 raft_consensus.cc:2388] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d1a3fde922040f493631610e6158ecf in current term 1: Already voted for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in this term.
I20250114 08:57:13.847507 20483 leader_election.cc:304] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf; no voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.848616 20718 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.856150 20664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:57918
I20250114 08:57:13.856849 20664 raft_consensus.cc:604] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:13.862864 20718 raft_consensus.cc:2743] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:13.863767 20666 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 2.
I20250114 08:57:13.865010 20483 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:13.865815 20718 raft_consensus.cc:2798] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:13.866276 20718 raft_consensus.cc:695] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:13.867116 20718 consensus_queue.cc:237] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.869316 20592 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 2.
I20250114 08:57:13.875643 20743 raft_consensus.cc:988] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: : Instructing follower 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:13.876086 20743 raft_consensus.cc:1076] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Signalling peer 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:13.879673 20743 raft_consensus.cc:988] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:13.881457 20743 raft_consensus.cc:1076] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:13.880566 20516 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:41682
I20250114 08:57:13.883648 20516 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:13.884507 20516 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.892431 20516 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.893596 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37948
I20250114 08:57:13.894276 20592 raft_consensus.cc:491] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:13.894670 20592 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.896971 20666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.897863 20666 raft_consensus.cc:3049] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:13.898260 20666 raft_consensus.cc:738] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.898944 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.899945 20591 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.901216 20743 consensus_queue.cc:1035] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:13.908841 20591 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 2.
I20250114 08:57:13.909902 20746 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.902307 20666 consensus_queue.cc:260] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.911082 20666 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.901288 20592 raft_consensus.cc:513] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.904875 20412 catalog_manager.cc:5526] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 1 to 2, leader changed from 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131) to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 2 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.910018 20483 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.918241 20718 raft_consensus.cc:2798] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:13.918967 20718 raft_consensus.cc:695] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:13.919471 20746 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:13.919881 20718 consensus_queue.cc:237] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.927001 20666 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 2.
I20250114 08:57:13.904616 20516 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 2 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.937579 20666 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:13.945498 20751 raft_consensus.cc:988] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:13.945966 20713 raft_consensus.cc:1076] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Signalling peer 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:13.947770 20514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a86050de79064e01a547176fae322bad"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:41668
I20250114 08:57:13.948343 20514 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:13.948616 20592 leader_election.cc:290] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.948782 20514 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.950253 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.950891 20665 raft_consensus.cc:3049] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:13.951210 20665 raft_consensus.cc:738] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:13.952041 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:13.952054 20665 consensus_queue.cc:260] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.952870 20517 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.953611 20665 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.956022 20514 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.959661 20516 raft_consensus.cc:1235] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Rejecting Update request from peer f6ccd6b41315457780eb97d4679f2b8b for earlier term 1. Current term is 2. Ops: []
I20250114 08:57:13.959707 20517 raft_consensus.cc:2463] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:13.960429 20665 raft_consensus.cc:2463] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:13.961162 20557 leader_election.cc:304] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:13.962157 20746 raft_consensus.cc:2798] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:13.962642 20746 raft_consensus.cc:695] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:13.963456 20746 consensus_queue.cc:237] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.966348 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:13.966845 20514 leader_election.cc:290] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 2 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:13.967021 20592 raft_consensus.cc:3049] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:13.967360 20592 raft_consensus.cc:738] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:13.967801 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:13.968418 20665 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.968094 20592 consensus_queue.cc:260] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.969483 20592 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:13.976001 20665 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 2.
I20250114 08:57:13.977268 20483 leader_election.cc:304] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:13.978138 20718 raft_consensus.cc:2798] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:13.977699 20411 catalog_manager.cc:5526] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 1 to 2, leader changed from 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131) to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 2 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.978597 20718 raft_consensus.cc:695] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:13.979447 20718 consensus_queue.cc:237] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:13.988694 20592 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 2.
I20250114 08:57:13.999178 20411 catalog_manager.cc:5526] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 1 to 2, leader changed from 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 2 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:13.999914 20413 catalog_manager.cc:5526] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 1 to 2, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 2 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:14.127285 20732 consensus_queue.cc:1035] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:14.133697 20718 raft_consensus.cc:988] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:14.134151 20718 raft_consensus.cc:1076] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:14.138870 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37934
I20250114 08:57:14.140033 20592 raft_consensus.cc:491] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:14.140489 20592 raft_consensus.cc:3054] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:14.142823 20718 consensus_queue.cc:1035] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:14.151826 20592 raft_consensus.cc:513] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:14.154757 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:14.155459 20516 raft_consensus.cc:3049] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:14.155886 20516 raft_consensus.cc:738] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:14.155723 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e3a9cff6d4f52b09fe03d5a1fb31f" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:14.156885 20665 raft_consensus.cc:3054] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:14.156924 20516 consensus_queue.cc:260] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:14.158300 20516 raft_consensus.cc:3054] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:14.155864 20592 leader_election.cc:290] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:14.163713 20516 raft_consensus.cc:2463] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:14.163727 20665 raft_consensus.cc:2463] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:14.164959 20557 leader_election.cc:304] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:14.165867 20746 raft_consensus.cc:2798] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:14.166332 20746 raft_consensus.cc:695] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:14.167141 20746 consensus_queue.cc:237] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:14.173894 20411 catalog_manager.cc:5526] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 1 to 2, leader changed from 4d1a3fde922040f493631610e6158ecf (127.19.231.129) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 2 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:14.345253 20516 raft_consensus.cc:1270] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.347293 20746 consensus_queue.cc:1035] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:57:14.356880 20665 raft_consensus.cc:1270] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.358680 20746 consensus_queue.cc:1035] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:57:14.371958 20732 raft_consensus.cc:988] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:14.372440 20718 raft_consensus.cc:1076] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:14.373808 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b616441962804786a716c5861c31876b"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37934
I20250114 08:57:14.374284 20592 raft_consensus.cc:491] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:14.374583 20592 raft_consensus.cc:3054] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:14.379472 20592 raft_consensus.cc:513] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:14.381444 20592 leader_election.cc:290] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:14.382329 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b616441962804786a716c5861c31876b" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:14.382655 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b616441962804786a716c5861c31876b" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:14.382948 20516 raft_consensus.cc:3049] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:14.383158 20665 raft_consensus.cc:3054] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:14.383283 20516 raft_consensus.cc:738] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:14.384054 20516 consensus_queue.cc:260] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:14.385288 20516 raft_consensus.cc:3054] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:14.387648 20665 raft_consensus.cc:2463] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:14.388610 20558 leader_election.cc:304] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:14.389250 20746 raft_consensus.cc:2798] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:14.389623 20746 raft_consensus.cc:695] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:14.390405 20746 consensus_queue.cc:237] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:14.391220 20516 raft_consensus.cc:2463] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:14.397368 20413 catalog_manager.cc:5526] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 1 to 2, leader changed from 4d1a3fde922040f493631610e6158ecf (127.19.231.129) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 2 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:14.412590 20592 raft_consensus.cc:1270] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.413836 20732 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.421454 20665 raft_consensus.cc:1270] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.423091 20732 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.430593 20665 raft_consensus.cc:1270] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.431836 20732 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.438817 20592 raft_consensus.cc:1270] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.440428 20731 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.461205 20665 raft_consensus.cc:1270] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.462430 20732 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.470283 20592 raft_consensus.cc:1270] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.471825 20718 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.570716 20665 raft_consensus.cc:1270] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.572077 20746 consensus_queue.cc:1035] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.580936 20514 raft_consensus.cc:1270] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.582746 20791 consensus_queue.cc:1035] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.965689 20516 raft_consensus.cc:1270] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.966811 20713 consensus_queue.cc:1035] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:14.976418 20665 raft_consensus.cc:1270] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:14.977617 20791 consensus_queue.cc:1035] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:16.890069 20382 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20250114 08:57:16.890849 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:16.911341 20516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a86050de79064e01a547176fae322bad"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
mode: GRACEFUL
new_leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:50732
I20250114 08:57:16.911880 20516 raft_consensus.cc:604] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Received request to transfer leadership to TS 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:16.922675 20805 raft_consensus.cc:988] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: : Instructing follower 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:16.922675 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:43386
I20250114 08:57:16.923288 20805 raft_consensus.cc:1076] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Signalling peer 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:16.923424 20592 raft_consensus.cc:604] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Received request to transfer leadership to TS 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:16.924630 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250114 08:57:16.924851 20665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a86050de79064e01a547176fae322bad"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:57912
I20250114 08:57:16.925227 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:16.925357 20665 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:16.925730 20665 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:16.930032 20665 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:16.931489 20665 leader_election.cc:290] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:16.932403 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:16.932495 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:16.933138 20516 raft_consensus.cc:3049] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:16.933162 20592 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:16.933626 20516 raft_consensus.cc:738] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:16.934444 20516 consensus_queue.cc:260] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:16.935686 20516 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:16.939780 20592 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 3.
I20250114 08:57:16.941004 20632 leader_election.cc:304] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:16.941740 20756 raft_consensus.cc:2798] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:16.941783 20516 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 3.
I20250114 08:57:16.943233 20756 raft_consensus.cc:695] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:16.944008 20756 consensus_queue.cc:237] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:16.950147 20413 catalog_manager.cc:5526] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 2 to 3, leader changed from 4d1a3fde922040f493631610e6158ecf (127.19.231.129) to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 3 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:17.291558 20813 raft_consensus.cc:988] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:17.291986 20813 raft_consensus.cc:1076] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Signalling peer 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:17.293489 20665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:57902
I20250114 08:57:17.293985 20665 raft_consensus.cc:491] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:17.294272 20665 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:17.298782 20665 raft_consensus.cc:513] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:17.300326 20665 leader_election.cc:290] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 2 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:17.301250 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:17.301401 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:17.301796 20516 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:17.301954 20592 raft_consensus.cc:3049] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:17.302264 20592 raft_consensus.cc:738] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:17.302867 20592 consensus_queue.cc:260] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:17.303763 20592 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:17.306329 20516 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 2.
I20250114 08:57:17.307322 20631 leader_election.cc:304] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:17.307766 20592 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 2.
I20250114 08:57:17.308017 20756 raft_consensus.cc:2798] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:17.308482 20756 raft_consensus.cc:695] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:17.309170 20756 consensus_queue.cc:237] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:17.318576 20413 catalog_manager.cc:5526] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 1 to 2, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:17.378531 20516 raft_consensus.cc:1270] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:17.379608 20756 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:17.389283 20592 raft_consensus.cc:1270] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:17.391453 20817 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:17.692385 20516 raft_consensus.cc:1270] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:17.693905 20817 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:17.701953 20592 raft_consensus.cc:1270] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:17.703207 20756 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:17.926240 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:17.931370 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:17.931775 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:18.932571 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:18.937415 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:18.937772 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:19.938922 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:19.945297 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:19.945703 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:20.946521 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:20.951119 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:20.951476 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:21.952150 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:21.957629 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:21.958065 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:22.959051 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:22.963832 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:22.964179 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:23.964979 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:23.970391 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:23.970803 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:24.971557 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:24.977200 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:24.977777 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:25.978571 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:25.983299 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:25.983726 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:27.008073 20382 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20250114 08:57:27.012189 20516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:41692
I20250114 08:57:27.012850 20516 raft_consensus.cc:604] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:27.017290 20516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:41692
I20250114 08:57:27.017731 20516 raft_consensus.cc:604] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:27.021821 20516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:41692
I20250114 08:57:27.022274 20516 raft_consensus.cc:604] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:27.026919 20665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a86050de79064e01a547176fae322bad"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:57918
I20250114 08:57:27.027343 20665 raft_consensus.cc:604] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:27.036006 20665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
mode: GRACEFUL
new_leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:57918
I20250114 08:57:27.036454 20665 raft_consensus.cc:604] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 LEADER]: Received request to transfer leadership to TS f6ccd6b41315457780eb97d4679f2b8b
I20250114 08:57:27.154618 20826 raft_consensus.cc:988] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.155133 20827 raft_consensus.cc:1076] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.156731 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37934
I20250114 08:57:27.157174 20592 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:27.157493 20592 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.162191 20592 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.163839 20592 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 3 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:27.164671 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:27.164944 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:27.165346 20516 raft_consensus.cc:3049] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:27.165474 20665 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.165649 20516 raft_consensus.cc:738] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:27.166312 20516 consensus_queue.cc:260] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.167241 20516 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.169909 20665 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 3.
I20250114 08:57:27.170995 20558 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:27.171515 20516 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 3.
I20250114 08:57:27.171747 20825 raft_consensus.cc:2798] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:27.172667 20825 raft_consensus.cc:695] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:27.173435 20825 consensus_queue.cc:237] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.180207 20411 catalog_manager.cc:5526] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 2 to 3, leader changed from 4d1a3fde922040f493631610e6158ecf (127.19.231.129) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 3 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:27.250358 20826 raft_consensus.cc:988] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.250771 20827 raft_consensus.cc:1076] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.252027 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37934
I20250114 08:57:27.252492 20592 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:27.252745 20592 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.256831 20592 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.258204 20592 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 3 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:27.258972 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:27.259183 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:27.259425 20516 raft_consensus.cc:3049] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:27.259617 20665 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.259718 20516 raft_consensus.cc:738] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:27.260299 20516 consensus_queue.cc:260] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.261229 20516 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.264366 20665 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 3.
I20250114 08:57:27.265262 20558 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:27.266005 20825 raft_consensus.cc:2798] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:27.266353 20825 raft_consensus.cc:695] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:27.266409 20516 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 3.
I20250114 08:57:27.267282 20825 consensus_queue.cc:237] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.273844 20411 catalog_manager.cc:5526] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 2 to 3, leader changed from 4d1a3fde922040f493631610e6158ecf (127.19.231.129) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 3 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:27.431529 20817 raft_consensus.cc:988] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.432027 20835 raft_consensus.cc:1076] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.433447 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37948
I20250114 08:57:27.433871 20592 raft_consensus.cc:491] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:27.434118 20592 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.438346 20592 raft_consensus.cc:513] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.440223 20817 raft_consensus.cc:988] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.440317 20592 leader_election.cc:290] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 3 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:27.440447 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:27.440721 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:27.440758 20817 raft_consensus.cc:1076] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.441403 20665 raft_consensus.cc:3049] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:27.441200 20516 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.441723 20665 raft_consensus.cc:738] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:27.442472 20665 consensus_queue.cc:260] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.442637 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a86050de79064e01a547176fae322bad"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37948
I20250114 08:57:27.443290 20592 raft_consensus.cc:491] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:27.443403 20665 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:27.443665 20592 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:27.447328 20665 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 3.
I20250114 08:57:27.447399 20516 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 3.
I20250114 08:57:27.448206 20558 leader_election.cc:304] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:27.449051 20825 raft_consensus.cc:2798] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:27.449541 20825 raft_consensus.cc:695] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:27.449344 20592 raft_consensus.cc:513] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.450465 20825 consensus_queue.cc:237] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.451565 20592 leader_election.cc:290] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 4 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:27.451925 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:27.455020 20516 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:27.452118 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86050de79064e01a547176fae322bad" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:27.455929 20665 raft_consensus.cc:3049] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:57:27.456274 20665 raft_consensus.cc:738] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:27.457149 20665 consensus_queue.cc:260] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.458375 20665 raft_consensus.cc:3054] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:27.461515 20516 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 4.
I20250114 08:57:27.461601 20411 catalog_manager.cc:5526] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 2 to 3, leader changed from 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 3 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:27.462877 20557 leader_election.cc:304] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:27.463716 20825 raft_consensus.cc:2798] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:57:27.464195 20825 raft_consensus.cc:695] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 4 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:27.464763 20665 raft_consensus.cc:2463] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 4.
I20250114 08:57:27.464920 20825 consensus_queue.cc:237] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.473409 20411 catalog_manager.cc:5526] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 3 to 4, leader changed from 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 4 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:27.481910 20826 raft_consensus.cc:988] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: : Instructing follower f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.482299 20827 raft_consensus.cc:1076] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Signalling peer f6ccd6b41315457780eb97d4679f2b8b to start an election
I20250114 08:57:27.483512 20592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
from {username='slave'} at 127.0.0.1:37934
I20250114 08:57:27.483923 20592 raft_consensus.cc:491] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:27.484163 20592 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:27.488020 20592 raft_consensus.cc:513] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.489425 20592 leader_election.cc:290] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:27.490191 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:27.490469 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "f6ccd6b41315457780eb97d4679f2b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:27.490777 20516 raft_consensus.cc:3049] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:27.490950 20665 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:27.491094 20516 raft_consensus.cc:738] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:27.491793 20516 consensus_queue.cc:260] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.492967 20516 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:27.496995 20665 raft_consensus.cc:2463] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:27.498000 20558 leader_election.cc:304] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fdaef96b2b44139ac68afc82bbf1d4, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:27.498797 20825 raft_consensus.cc:2798] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:27.498807 20516 raft_consensus.cc:2463] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ccd6b41315457780eb97d4679f2b8b in term 2.
I20250114 08:57:27.499258 20825 raft_consensus.cc:695] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Becoming Leader. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:27.500114 20825 consensus_queue.cc:237] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:27.506250 20411 catalog_manager.cc:5526] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b reported cstate change: term changed from 1 to 2, leader changed from 4d1a3fde922040f493631610e6158ecf (127.19.231.129) to f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130). New cstate: current_term: 2 leader_uuid: "f6ccd6b41315457780eb97d4679f2b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:27.577374 20516 raft_consensus.cc:1270] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:27.578464 20825 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:27.588902 20665 raft_consensus.cc:1270] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:27.590703 20842 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:57:27.665141 20516 raft_consensus.cc:1270] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:27.666267 20842 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:27.674489 20665 raft_consensus.cc:1270] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:27.676067 20825 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:27.914933 20516 raft_consensus.cc:1270] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:27.916249 20825 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:27.926255 20665 raft_consensus.cc:1270] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:27.927472 20828 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:57:27.950179 20665 raft_consensus.cc:1270] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:27.951316 20828 consensus_queue.cc:1035] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:27.960800 20516 raft_consensus.cc:1270] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:27.962447 20825 consensus_queue.cc:1035] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:28.036267 20516 raft_consensus.cc:1270] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:28.037372 20825 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:28.045897 20665 raft_consensus.cc:1270] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Refusing update from remote peer f6ccd6b41315457780eb97d4679f2b8b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:28.047201 20825 consensus_queue.cc:1035] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:30.070981 20382 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20250114 08:57:30.071735 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:30.080279 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:43386
I20250114 08:57:30.081099 20592 raft_consensus.cc:604] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Received request to transfer leadership to TS 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:30.083766 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:43386
I20250114 08:57:30.084210 20592 raft_consensus.cc:604] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Received request to transfer leadership to TS 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:30.086339 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:43386
I20250114 08:57:30.086788 20592 raft_consensus.cc:604] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Received request to transfer leadership to TS 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:30.088796 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:43386
I20250114 08:57:30.089216 20592 raft_consensus.cc:604] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Received request to transfer leadership to TS 95fdaef96b2b44139ac68afc82bbf1d4
I20250114 08:57:30.091363 20592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89"
dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
mode: GRACEFUL
new_leader_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:43386
I20250114 08:57:30.091964 20592 raft_consensus.cc:604] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Received request to transfer leadership to TS 4d1a3fde922040f493631610e6158ecf
I20250114 08:57:30.093029 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:57:30.093700 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:30.102020 20854 raft_consensus.cc:988] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:30.102483 20854 raft_consensus.cc:1076] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Signalling peer 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:30.103912 20514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:41668
I20250114 08:57:30.104375 20514 raft_consensus.cc:491] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:30.104751 20514 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.116899 20514 raft_consensus.cc:513] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.119247 20514 leader_election.cc:290] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 4 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:30.120411 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:30.120584 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba72626b63a421c9be0e746049b07a5" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:30.121122 20592 raft_consensus.cc:3049] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:57:30.121234 20665 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.121412 20592 raft_consensus.cc:738] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:30.122119 20592 consensus_queue.cc:260] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.123310 20592 raft_consensus.cc:3054] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.129253 20848 raft_consensus.cc:988] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:30.129683 20854 raft_consensus.cc:1076] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Signalling peer 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:30.130981 20514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:41668
I20250114 08:57:30.131444 20514 raft_consensus.cc:491] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:30.131738 20514 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:30.180313 20592 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 4.
I20250114 08:57:30.180402 20665 raft_consensus.cc:2463] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 4.
I20250114 08:57:30.180419 20514 raft_consensus.cc:513] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.182193 20483 leader_election.cc:304] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:30.183162 20514 leader_election.cc:290] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 3 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:30.183966 20857 raft_consensus.cc:2798] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:57:30.185295 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:30.185297 20857 raft_consensus.cc:695] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 4 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:30.185985 20665 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:30.185297 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b34b35aa1ef46f79000df66f3375c34" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:30.186651 20592 raft_consensus.cc:3049] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:30.187018 20592 raft_consensus.cc:738] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:30.186576 20857 consensus_queue.cc:237] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.187765 20592 consensus_queue.cc:260] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.189059 20592 raft_consensus.cc:3054] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:30.195183 20413 catalog_manager.cc:5526] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 3 to 4, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 4 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:30.220880 20665 raft_consensus.cc:2463] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 3.
I20250114 08:57:30.220885 20592 raft_consensus.cc:2463] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 3.
I20250114 08:57:30.222230 20483 leader_election.cc:304] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:30.223232 20857 raft_consensus.cc:2798] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:30.223660 20857 raft_consensus.cc:695] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 3 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:30.224530 20857 consensus_queue.cc:237] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.230930 20413 catalog_manager.cc:5526] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 2 to 3, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 3 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:30.307154 20848 raft_consensus.cc:988] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:30.307579 20854 raft_consensus.cc:1076] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Signalling peer 4d1a3fde922040f493631610e6158ecf to start an election
I20250114 08:57:30.308986 20514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89"
dest_uuid: "4d1a3fde922040f493631610e6158ecf"
from {username='slave'} at 127.0.0.1:41668
I20250114 08:57:30.309449 20514 raft_consensus.cc:491] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:30.309710 20514 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.318529 20514 raft_consensus.cc:513] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.320178 20514 leader_election.cc:290] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 4 election: Requested vote from peers f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569), 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131:44967)
I20250114 08:57:30.321130 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:30.321336 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cafaca6a7944139c6fab5add1daf89" candidate_uuid: "4d1a3fde922040f493631610e6158ecf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
I20250114 08:57:30.321853 20665 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.321781 20592 raft_consensus.cc:3049] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:57:30.322300 20592 raft_consensus.cc:738] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:30.322957 20592 consensus_queue.cc:260] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.323834 20592 raft_consensus.cc:3054] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.357311 20848 raft_consensus.cc:988] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:30.357774 20854 raft_consensus.cc:1076] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Signalling peer 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:30.359172 20666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:57902
I20250114 08:57:30.359601 20666 raft_consensus.cc:491] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:30.359891 20666 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:30.420384 20665 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 4.
I20250114 08:57:30.420384 20592 raft_consensus.cc:2463] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1a3fde922040f493631610e6158ecf in term 4.
I20250114 08:57:30.422111 20483 leader_election.cc:304] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, f6ccd6b41315457780eb97d4679f2b8b; no voters:
I20250114 08:57:30.422919 20857 raft_consensus.cc:2798] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:57:30.423429 20857 raft_consensus.cc:695] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 4 LEADER]: Becoming Leader. State: Replica: 4d1a3fde922040f493631610e6158ecf, State: Running, Role: LEADER
I20250114 08:57:30.424430 20857 consensus_queue.cc:237] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.430644 20413 catalog_manager.cc:5526] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf reported cstate change: term changed from 3 to 4, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 4d1a3fde922040f493631610e6158ecf (127.19.231.129). New cstate: current_term: 4 leader_uuid: "4d1a3fde922040f493631610e6158ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:30.475353 20848 raft_consensus.cc:988] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: : Instructing follower 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:30.475773 20854 raft_consensus.cc:1076] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Signalling peer 95fdaef96b2b44139ac68afc82bbf1d4 to start an election
I20250114 08:57:30.477217 20665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8"
dest_uuid: "95fdaef96b2b44139ac68afc82bbf1d4"
from {username='slave'} at 127.0.0.1:57902
I20250114 08:57:30.477681 20665 raft_consensus.cc:491] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:30.477916 20665 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.519409 20666 raft_consensus.cc:513] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.521750 20666 leader_election.cc:290] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:30.523000 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:30.523334 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a627ee6253c849afaf6ac1f6e82ab6e3" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:30.523589 20514 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:30.524060 20592 raft_consensus.cc:3049] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:30.524375 20592 raft_consensus.cc:738] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:30.525053 20592 consensus_queue.cc:260] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.525959 20592 raft_consensus.cc:3054] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:30.582641 20591 raft_consensus.cc:1270] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:30.583895 20857 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:30.597180 20666 raft_consensus.cc:1270] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:30.598862 20858 consensus_queue.cc:1035] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:30.617381 20514 raft_consensus.cc:2463] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 3.
I20250114 08:57:30.617383 20592 raft_consensus.cc:2463] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 3.
I20250114 08:57:30.617409 20665 raft_consensus.cc:513] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.619014 20631 leader_election.cc:304] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:30.619556 20665 leader_election.cc:290] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 4 election: Requested vote from peers 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823), f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569)
I20250114 08:57:30.620633 20855 raft_consensus.cc:2798] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:30.620950 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6ccd6b41315457780eb97d4679f2b8b"
I20250114 08:57:30.621568 20592 raft_consensus.cc:3049] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:57:30.621850 20592 raft_consensus.cc:738] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6ccd6b41315457780eb97d4679f2b8b, State: Running, Role: LEADER
I20250114 08:57:30.622138 20855 raft_consensus.cc:695] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:30.622248 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8abd09fe96b445c93371b0eb694cee8" candidate_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d1a3fde922040f493631610e6158ecf"
I20250114 08:57:30.622851 20514 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.622545 20592 consensus_queue.cc:260] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.623085 20855 consensus_queue.cc:237] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.623718 20592 raft_consensus.cc:3054] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:30.631304 20414 catalog_manager.cc:5526] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 2 to 3, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 3 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:30.717056 20665 raft_consensus.cc:1270] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:30.718218 20858 consensus_queue.cc:1035] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:30.721602 20592 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 4.
I20250114 08:57:30.721674 20514 raft_consensus.cc:2463] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fdaef96b2b44139ac68afc82bbf1d4 in term 4.
I20250114 08:57:30.723371 20631 leader_election.cc:304] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1a3fde922040f493631610e6158ecf, 95fdaef96b2b44139ac68afc82bbf1d4; no voters:
I20250114 08:57:30.724429 20855 raft_consensus.cc:2798] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:57:30.725186 20855 raft_consensus.cc:695] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 LEADER]: Becoming Leader. State: Replica: 95fdaef96b2b44139ac68afc82bbf1d4, State: Running, Role: LEADER
I20250114 08:57:30.726090 20855 consensus_queue.cc:237] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } }
I20250114 08:57:30.729591 20592 raft_consensus.cc:1270] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:30.731340 20869 consensus_queue.cc:1035] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:57:30.738938 20413 catalog_manager.cc:5526] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 reported cstate change: term changed from 3 to 4, leader changed from f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130) to 95fdaef96b2b44139ac68afc82bbf1d4 (127.19.231.131). New cstate: current_term: 4 leader_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:31.014410 20592 raft_consensus.cc:1270] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:31.015659 20857 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:31.024852 20665 raft_consensus.cc:1270] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Refusing update from remote peer 4d1a3fde922040f493631610e6158ecf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:31.026358 20858 consensus_queue.cc:1035] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fdaef96b2b44139ac68afc82bbf1d4" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:31.094596 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:31.101729 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:31.102311 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:31.156965 20592 raft_consensus.cc:1270] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:31.158159 20855 consensus_queue.cc:1035] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:31.167321 20514 raft_consensus.cc:1270] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:31.168709 20882 consensus_queue.cc:1035] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:57:31.243664 20516 raft_consensus.cc:1270] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:31.244889 20882 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1a3fde922040f493631610e6158ecf" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:31.254148 20591 raft_consensus.cc:1270] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Refusing update from remote peer 95fdaef96b2b44139ac68afc82bbf1d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:31.255802 20885 consensus_queue.cc:1035] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ccd6b41315457780eb97d4679f2b8b" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:57:32.103171 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:32.108397 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:32.108857 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:33.109700 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:33.115736 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:33.116281 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:34.117156 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:34.121910 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:34.122336 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:35.123163 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:35.128161 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:35.128620 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:36.129458 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:36.134219 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:36.134661 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:37.135564 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:37.140723 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:37.141198 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:38.141942 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:38.147022 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:38.147490 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:39.148308 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:39.153160 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:39.153604 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:40.182214 20382 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20250114 08:57:40.182879 20382 tablet_server.cc:178] TabletServer@127.19.231.129:0 shutting down...
W20250114 08:57:40.188547 20631 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250114 08:57:40.194025 20631 consensus_peers.cc:487] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 -> Peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Couldn't send request to peer 4d1a3fde922040f493631610e6158ecf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:57:40.211802 20382 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 08:57:40.212648 20382 tablet_replica.cc:331] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.213418 20382 raft_consensus.cc:2238] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.214082 20382 raft_consensus.cc:2267] T a627ee6253c849afaf6ac1f6e82ab6e3 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.216624 20382 tablet_replica.cc:331] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.217146 20382 raft_consensus.cc:2238] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.217576 20382 raft_consensus.cc:2267] T fa033b09208d4422b4a55b35f4835ff1 P 4d1a3fde922040f493631610e6158ecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.219267 20382 tablet_replica.cc:331] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.219748 20382 raft_consensus.cc:2238] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 4 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.220772 20382 raft_consensus.cc:2267] T f3cafaca6a7944139c6fab5add1daf89 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.222525 20382 tablet_replica.cc:331] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.223028 20382 raft_consensus.cc:2238] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.223533 20382 raft_consensus.cc:2267] T d8abd09fe96b445c93371b0eb694cee8 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.225241 20382 tablet_replica.cc:331] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.225752 20382 raft_consensus.cc:2238] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 4 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.226697 20382 raft_consensus.cc:2267] T aba72626b63a421c9be0e746049b07a5 P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.228557 20382 tablet_replica.cc:331] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.229063 20382 raft_consensus.cc:2238] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.229658 20382 raft_consensus.cc:2267] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.231601 20382 tablet_replica.cc:331] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.232069 20382 raft_consensus.cc:2238] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.232527 20382 raft_consensus.cc:2267] T b616441962804786a716c5861c31876b P 4d1a3fde922040f493631610e6158ecf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.234684 20382 tablet_replica.cc:331] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.235167 20382 raft_consensus.cc:2238] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.236295 20382 raft_consensus.cc:2267] T 3b34b35aa1ef46f79000df66f3375c34 P 4d1a3fde922040f493631610e6158ecf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.238296 20382 tablet_replica.cc:331] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf: stopping tablet replica
I20250114 08:57:40.238742 20382 raft_consensus.cc:2238] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.239183 20382 raft_consensus.cc:2267] T a86050de79064e01a547176fae322bad P 4d1a3fde922040f493631610e6158ecf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.329607 20382 tablet_server.cc:195] TabletServer@127.19.231.129:0 shutdown complete.
W20250114 08:57:40.331076 20557 consensus_peers.cc:487] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b -> Peer 4d1a3fde922040f493631610e6158ecf (127.19.231.129:35823): Couldn't send request to peer 4d1a3fde922040f493631610e6158ecf. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.129:35823: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 08:57:40.348433 20382 tablet_server.cc:178] TabletServer@127.19.231.130:0 shutting down...
I20250114 08:57:40.375771 20382 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 08:57:40.376447 20382 tablet_replica.cc:331] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.377063 20382 raft_consensus.cc:2238] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.377538 20382 raft_consensus.cc:2267] T fa033b09208d4422b4a55b35f4835ff1 P f6ccd6b41315457780eb97d4679f2b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.379366 20382 tablet_replica.cc:331] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.379870 20382 raft_consensus.cc:2238] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.380311 20382 raft_consensus.cc:2267] T a627ee6253c849afaf6ac1f6e82ab6e3 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.382018 20382 tablet_replica.cc:331] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.382511 20382 raft_consensus.cc:2238] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 4 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.383335 20382 raft_consensus.cc:2267] T a86050de79064e01a547176fae322bad P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.385028 20382 tablet_replica.cc:331] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.385497 20382 raft_consensus.cc:2238] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.385886 20382 raft_consensus.cc:2267] T 3b34b35aa1ef46f79000df66f3375c34 P f6ccd6b41315457780eb97d4679f2b8b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.387498 20382 tablet_replica.cc:331] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.387979 20382 raft_consensus.cc:2238] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.388839 20382 raft_consensus.cc:2267] T b616441962804786a716c5861c31876b P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.390479 20382 tablet_replica.cc:331] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.390914 20382 raft_consensus.cc:2238] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 2 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.391646 20382 raft_consensus.cc:2267] T 998e3a9cff6d4f52b09fe03d5a1fb31f P f6ccd6b41315457780eb97d4679f2b8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.393260 20382 tablet_replica.cc:331] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.393764 20382 raft_consensus.cc:2238] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.394187 20382 raft_consensus.cc:2267] T aba72626b63a421c9be0e746049b07a5 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.395788 20382 tablet_replica.cc:331] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.396214 20382 raft_consensus.cc:2238] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.396651 20382 raft_consensus.cc:2267] T d8abd09fe96b445c93371b0eb694cee8 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.398283 20382 tablet_replica.cc:331] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b: stopping tablet replica
I20250114 08:57:40.398721 20382 raft_consensus.cc:2238] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.399124 20382 raft_consensus.cc:2267] T f3cafaca6a7944139c6fab5add1daf89 P f6ccd6b41315457780eb97d4679f2b8b [term 4 FOLLOWER]: Raft consensus is shut down!
W20250114 08:57:40.402271 20632 consensus_peers.cc:487] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 -> Peer f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569): Couldn't send request to peer f6ccd6b41315457780eb97d4679f2b8b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:57:40.428350 20382 tablet_server.cc:195] TabletServer@127.19.231.130:0 shutdown complete.
I20250114 08:57:40.446157 20382 tablet_server.cc:178] TabletServer@127.19.231.131:0 shutting down...
W20250114 08:57:40.459942 20632 consensus_peers.cc:487] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 -> Peer f6ccd6b41315457780eb97d4679f2b8b (127.19.231.130:33569): Couldn't send request to peer f6ccd6b41315457780eb97d4679f2b8b. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.130:33569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 08:57:40.474040 20382 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 08:57:40.474767 20382 tablet_replica.cc:331] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.475389 20382 raft_consensus.cc:2238] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.475888 20382 raft_consensus.cc:2267] T 3b34b35aa1ef46f79000df66f3375c34 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.477669 20382 tablet_replica.cc:331] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.478175 20382 raft_consensus.cc:2238] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.478617 20382 raft_consensus.cc:2267] T a86050de79064e01a547176fae322bad P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.480253 20382 tablet_replica.cc:331] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.480831 20382 raft_consensus.cc:2238] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.481666 20382 raft_consensus.cc:2267] T d8abd09fe96b445c93371b0eb694cee8 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.483250 20382 tablet_replica.cc:331] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.483736 20382 raft_consensus.cc:2238] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.484586 20382 raft_consensus.cc:2267] T a627ee6253c849afaf6ac1f6e82ab6e3 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.486194 20382 tablet_replica.cc:331] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.486685 20382 raft_consensus.cc:2238] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.487092 20382 raft_consensus.cc:2267] T b616441962804786a716c5861c31876b P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.488813 20382 tablet_replica.cc:331] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.489288 20382 raft_consensus.cc:2238] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.489711 20382 raft_consensus.cc:2267] T 998e3a9cff6d4f52b09fe03d5a1fb31f P 95fdaef96b2b44139ac68afc82bbf1d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.491304 20382 tablet_replica.cc:331] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.491750 20382 raft_consensus.cc:2238] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.492159 20382 raft_consensus.cc:2267] T aba72626b63a421c9be0e746049b07a5 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.494061 20382 tablet_replica.cc:331] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.494494 20382 raft_consensus.cc:2238] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:57:40.494998 20382 raft_consensus.cc:2267] T f3cafaca6a7944139c6fab5add1daf89 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.496690 20382 tablet_replica.cc:331] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4: stopping tablet replica
I20250114 08:57:40.497116 20382 raft_consensus.cc:2238] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.497895 20382 raft_consensus.cc:2267] T fa033b09208d4422b4a55b35f4835ff1 P 95fdaef96b2b44139ac68afc82bbf1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.527184 20382 tablet_server.cc:195] TabletServer@127.19.231.131:0 shutdown complete.
I20250114 08:57:40.545611 20382 master.cc:537] Master@127.19.231.190:40473 shutting down...
I20250114 08:57:40.721737 20382 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 1 LEADER]: Raft consensus shutting down.
I20250114 08:57:40.722395 20382 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:57:40.722728 20382 tablet_replica.cc:331] T 00000000000000000000000000000000 P 910dfd3b30cd4d1095fedb11d8f6fb46: stopping tablet replica
I20250114 08:57:40.742705 20382 master.cc:559] Master@127.19.231.190:40473 shutdown complete.
4d1a3fde922040f493631610e6158ecf 4
95fdaef96b2b44139ac68afc82bbf1d4 1
f6ccd6b41315457780eb97d4679f2b8b 4
4d1a3fde922040f493631610e6158ecf 3
95fdaef96b2b44139ac68afc82bbf1d4 3
f6ccd6b41315457780eb97d4679f2b8b 3
4d1a3fde922040f493631610e6158ecf 0
95fdaef96b2b44139ac68afc82bbf1d4 1
f6ccd6b41315457780eb97d4679f2b8b 8
4d1a3fde922040f493631610e6158ecf 3
95fdaef96b2b44139ac68afc82bbf1d4 3
f6ccd6b41315457780eb97d4679f2b8b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30969 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250114 08:57:40.783761 20382 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.231.190:45087
I20250114 08:57:40.784973 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:40.790939 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:40.792515 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:40.793599 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:40.793828 20382 server_base.cc:1034] running on GCE node
I20250114 08:57:40.794926 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:40.795131 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:40.795287 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845060795272 us; error 0 us; skew 500 ppm
I20250114 08:57:40.795841 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:40.798421 20382 webserver.cc:458] Webserver started at http://127.19.231.190:42067/ using document root <none> and password file <none>
I20250114 08:57:40.798956 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:40.799144 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:40.799407 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:40.800920 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/master-0-root/instance:
uuid: "28b0eb2db0c5487c9e24094027020b29"
format_stamp: "Formatted at 2025-01-14 08:57:40 on dist-test-slave-4wc2"
I20250114 08:57:40.805718 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250114 08:57:40.809114 20906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:40.809999 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250114 08:57:40.810263 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/master-0-root
uuid: "28b0eb2db0c5487c9e24094027020b29"
format_stamp: "Formatted at 2025-01-14 08:57:40 on dist-test-slave-4wc2"
I20250114 08:57:40.810516 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:40.823370 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:40.824697 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:40.864329 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.190:45087
I20250114 08:57:40.864442 20957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.190:45087 every 8 connection(s)
I20250114 08:57:40.868346 20958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:40.878695 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29: Bootstrap starting.
I20250114 08:57:40.883416 20958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:40.887712 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29: No bootstrap required, opened a new log
I20250114 08:57:40.889811 20958 raft_consensus.cc:357] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b0eb2db0c5487c9e24094027020b29" member_type: VOTER }
I20250114 08:57:40.890303 20958 raft_consensus.cc:383] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:40.890547 20958 raft_consensus.cc:738] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b0eb2db0c5487c9e24094027020b29, State: Initialized, Role: FOLLOWER
I20250114 08:57:40.891091 20958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b0eb2db0c5487c9e24094027020b29" member_type: VOTER }
I20250114 08:57:40.891535 20958 raft_consensus.cc:397] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 08:57:40.891788 20958 raft_consensus.cc:491] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 08:57:40.892057 20958 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:40.896070 20958 raft_consensus.cc:513] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b0eb2db0c5487c9e24094027020b29" member_type: VOTER }
I20250114 08:57:40.896654 20958 leader_election.cc:304] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b0eb2db0c5487c9e24094027020b29; no voters:
I20250114 08:57:40.897791 20958 leader_election.cc:290] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 08:57:40.898198 20961 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:40.899550 20961 raft_consensus.cc:695] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 1 LEADER]: Becoming Leader. State: Replica: 28b0eb2db0c5487c9e24094027020b29, State: Running, Role: LEADER
I20250114 08:57:40.900295 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b0eb2db0c5487c9e24094027020b29" member_type: VOTER }
I20250114 08:57:40.900830 20958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 08:57:40.905124 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b0eb2db0c5487c9e24094027020b29. Latest consensus state: current_term: 1 leader_uuid: "28b0eb2db0c5487c9e24094027020b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b0eb2db0c5487c9e24094027020b29" member_type: VOTER } }
I20250114 08:57:40.905777 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [sys.catalog]: This master's current role is: LEADER
I20250114 08:57:40.905987 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b0eb2db0c5487c9e24094027020b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b0eb2db0c5487c9e24094027020b29" member_type: VOTER } }
I20250114 08:57:40.906535 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [sys.catalog]: This master's current role is: LEADER
I20250114 08:57:40.908820 20967 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 08:57:40.917315 20967 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 08:57:40.918327 20382 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 08:57:40.925940 20967 catalog_manager.cc:1348] Generated new cluster ID: e323ce892aed405fb3b81e918dc88c73
I20250114 08:57:40.926252 20967 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 08:57:40.941367 20967 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 08:57:40.942616 20967 catalog_manager.cc:1505] Loading token signing keys...
I20250114 08:57:40.965612 20967 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29: Generated new TSK 0
I20250114 08:57:40.966233 20967 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 08:57:40.985388 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:40.991573 20979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:40.993155 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:40.994159 20382 server_base.cc:1034] running on GCE node
W20250114 08:57:40.994954 20982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:40.995813 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:40.995999 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:40.996122 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845060996112 us; error 0 us; skew 500 ppm
I20250114 08:57:40.996647 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:41.003857 20382 webserver.cc:458] Webserver started at http://127.19.231.129:38657/ using document root <none> and password file <none>
I20250114 08:57:41.004357 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:41.004566 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:41.004846 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:41.005900 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-0-root/instance:
uuid: "26d40bfe96134eb0b1478d2664b2086e"
format_stamp: "Formatted at 2025-01-14 08:57:41 on dist-test-slave-4wc2"
I20250114 08:57:41.010411 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250114 08:57:41.013808 20987 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.014540 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 08:57:41.014832 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-0-root
uuid: "26d40bfe96134eb0b1478d2664b2086e"
format_stamp: "Formatted at 2025-01-14 08:57:41 on dist-test-slave-4wc2"
I20250114 08:57:41.015089 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:41.024689 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:41.025909 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:41.027360 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:57:41.029680 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:57:41.029884 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.030113 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:57:41.030292 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.070941 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.129:36059
I20250114 08:57:41.071033 21049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.129:36059 every 8 connection(s)
I20250114 08:57:41.075924 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:41.083526 21054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:41.084969 21055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:41.087159 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:41.088438 20382 server_base.cc:1034] running on GCE node
I20250114 08:57:41.089138 21050 heartbeater.cc:346] Connected to a master server at 127.19.231.190:45087
I20250114 08:57:41.089610 21050 heartbeater.cc:463] Registering TS with master...
I20250114 08:57:41.090322 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:41.090605 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:41.090571 21050 heartbeater.cc:510] Master 127.19.231.190:45087 requested a full tablet report, sending...
I20250114 08:57:41.090909 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845061090890 us; error 0 us; skew 500 ppm
I20250114 08:57:41.091641 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:41.093155 20923 ts_manager.cc:194] Registered new tserver with Master: 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059)
I20250114 08:57:41.094501 20382 webserver.cc:458] Webserver started at http://127.19.231.130:35055/ using document root <none> and password file <none>
I20250114 08:57:41.095017 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:41.094995 20923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37894
I20250114 08:57:41.095360 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:41.095685 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:41.097101 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-1-root/instance:
uuid: "08a5ac200b51404694fa174ebc6fe0d5"
format_stamp: "Formatted at 2025-01-14 08:57:41 on dist-test-slave-4wc2"
I20250114 08:57:41.102202 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250114 08:57:41.105636 21062 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.106436 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 08:57:41.106691 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-1-root
uuid: "08a5ac200b51404694fa174ebc6fe0d5"
format_stamp: "Formatted at 2025-01-14 08:57:41 on dist-test-slave-4wc2"
I20250114 08:57:41.106935 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:41.124404 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:41.125674 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:41.127106 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:57:41.129346 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:57:41.129555 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.129788 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:57:41.129942 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.186043 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.130:45771
I20250114 08:57:41.186090 21124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.130:45771 every 8 connection(s)
I20250114 08:57:41.192961 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:57:41.199636 21128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:41.202353 21129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:57:41.204094 21132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:57:41.205673 21125 heartbeater.cc:346] Connected to a master server at 127.19.231.190:45087
I20250114 08:57:41.205802 20382 server_base.cc:1034] running on GCE node
I20250114 08:57:41.206270 21125 heartbeater.cc:463] Registering TS with master...
I20250114 08:57:41.207136 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:57:41.207441 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:57:41.207360 21125 heartbeater.cc:510] Master 127.19.231.190:45087 requested a full tablet report, sending...
I20250114 08:57:41.207731 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845061207711 us; error 0 us; skew 500 ppm
I20250114 08:57:41.208559 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:57:41.210024 20923 ts_manager.cc:194] Registered new tserver with Master: 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:41.211552 20382 webserver.cc:458] Webserver started at http://127.19.231.131:43313/ using document root <none> and password file <none>
I20250114 08:57:41.211946 20923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37904
I20250114 08:57:41.212239 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:57:41.212517 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:57:41.212915 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:57:41.214111 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-2-root/instance:
uuid: "8905c307b2d04181a6333a806893cbdc"
format_stamp: "Formatted at 2025-01-14 08:57:41 on dist-test-slave-4wc2"
I20250114 08:57:41.218582 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20250114 08:57:41.221835 21136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.222565 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 08:57:41.222862 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-2-root
uuid: "8905c307b2d04181a6333a806893cbdc"
format_stamp: "Formatted at 2025-01-14 08:57:41 on dist-test-slave-4wc2"
I20250114 08:57:41.223141 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736845029639699-20382-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:57:41.240918 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:57:41.242316 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:57:41.243796 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:57:41.246070 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:57:41.246282 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.246527 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:57:41.246685 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:57:41.286685 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.131:40787
I20250114 08:57:41.286792 21198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.131:40787 every 8 connection(s)
I20250114 08:57:41.300561 21199 heartbeater.cc:346] Connected to a master server at 127.19.231.190:45087
I20250114 08:57:41.300951 21199 heartbeater.cc:463] Registering TS with master...
I20250114 08:57:41.301697 21199 heartbeater.cc:510] Master 127.19.231.190:45087 requested a full tablet report, sending...
I20250114 08:57:41.303603 20923 ts_manager.cc:194] Registered new tserver with Master: 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.304661 20382 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014767712s
I20250114 08:57:41.305116 20923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37916
I20250114 08:57:41.307055 20382 test_util.cc:274] Using random seed: -1058489708
I20250114 08:57:41.331866 20923 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37924:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 08:57:41.334940 20923 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 08:57:41.415673 21015 tablet_service.cc:1467] Processing CreateTablet for tablet 8e23fe994c2d4a818f29a1c9c4caf4c3 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 08:57:41.416417 21164 tablet_service.cc:1467] Processing CreateTablet for tablet 8e23fe994c2d4a818f29a1c9c4caf4c3 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 08:57:41.417181 21015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e23fe994c2d4a818f29a1c9c4caf4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.417814 21164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e23fe994c2d4a818f29a1c9c4caf4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.417964 21009 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1e12f111846c7968bcd95be8ebc5f (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 08:57:41.418937 21006 tablet_service.cc:1467] Processing CreateTablet for tablet 11dd206a2b6246a1935e5b943ab23baf (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 08:57:41.419456 21009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d1e12f111846c7968bcd95be8ebc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.419508 21163 tablet_service.cc:1467] Processing CreateTablet for tablet 060a086354884b0786d31a6d67b07f81 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 08:57:41.420502 21006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11dd206a2b6246a1935e5b943ab23baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.421133 21163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060a086354884b0786d31a6d67b07f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.421707 21013 tablet_service.cc:1467] Processing CreateTablet for tablet c8d5ba27eb7e4d2e9aea1fff2b0ab1ae (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 08:57:41.423015 21013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d5ba27eb7e4d2e9aea1fff2b0ab1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.425376 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 060a086354884b0786d31a6d67b07f81 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 08:57:41.425691 21012 tablet_service.cc:1467] Processing CreateTablet for tablet 13273a9f79d14d13b2569c8e082f7a11 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 08:57:41.426733 21014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060a086354884b0786d31a6d67b07f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.428099 21011 tablet_service.cc:1467] Processing CreateTablet for tablet a9c51502140b4ab691481d101afb0f6c (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 08:57:41.428910 21012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13273a9f79d14d13b2569c8e082f7a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.434715 21007 tablet_service.cc:1467] Processing CreateTablet for tablet f559c3fc5ad7468f9b13ba6572cc25cd (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 08:57:41.436149 21007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f559c3fc5ad7468f9b13ba6572cc25cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.437835 21008 tablet_service.cc:1467] Processing CreateTablet for tablet a69b04a506c84abda09636ce474bb7ce (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 08:57:41.440269 21010 tablet_service.cc:1467] Processing CreateTablet for tablet 707dcb1fdaf64a2e9f926b3c0c746c55 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 08:57:41.442617 21162 tablet_service.cc:1467] Processing CreateTablet for tablet c8d5ba27eb7e4d2e9aea1fff2b0ab1ae (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 08:57:41.444005 21162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d5ba27eb7e4d2e9aea1fff2b0ab1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.445442 21156 tablet_service.cc:1467] Processing CreateTablet for tablet f559c3fc5ad7468f9b13ba6572cc25cd (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 08:57:41.445649 21161 tablet_service.cc:1467] Processing CreateTablet for tablet 13273a9f79d14d13b2569c8e082f7a11 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 08:57:41.446899 21156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f559c3fc5ad7468f9b13ba6572cc25cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.447547 21160 tablet_service.cc:1467] Processing CreateTablet for tablet a9c51502140b4ab691481d101afb0f6c (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 08:57:41.449584 21159 tablet_service.cc:1467] Processing CreateTablet for tablet 707dcb1fdaf64a2e9f926b3c0c746c55 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 08:57:41.450104 21155 tablet_service.cc:1467] Processing CreateTablet for tablet 11dd206a2b6246a1935e5b943ab23baf (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 08:57:41.451613 21158 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1e12f111846c7968bcd95be8ebc5f (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 08:57:41.453650 21157 tablet_service.cc:1467] Processing CreateTablet for tablet a69b04a506c84abda09636ce474bb7ce (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 08:57:41.455092 21160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c51502140b4ab691481d101afb0f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.458189 21011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c51502140b4ab691481d101afb0f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.459085 21008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69b04a506c84abda09636ce474bb7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.460379 21161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13273a9f79d14d13b2569c8e082f7a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.461058 21155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11dd206a2b6246a1935e5b943ab23baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.465117 21158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d1e12f111846c7968bcd95be8ebc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.469049 21157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69b04a506c84abda09636ce474bb7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.461084 21010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707dcb1fdaf64a2e9f926b3c0c746c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.473182 21214 tablet_bootstrap.cc:492] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.476644 21159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707dcb1fdaf64a2e9f926b3c0c746c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.517088 21214 tablet_bootstrap.cc:654] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.549842 21215 tablet_bootstrap.cc:492] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.554641 21086 tablet_service.cc:1467] Processing CreateTablet for tablet a9c51502140b4ab691481d101afb0f6c (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 08:57:41.555228 21088 tablet_service.cc:1467] Processing CreateTablet for tablet c8d5ba27eb7e4d2e9aea1fff2b0ab1ae (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 08:57:41.556203 21086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c51502140b4ab691481d101afb0f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.554656 21087 tablet_service.cc:1467] Processing CreateTablet for tablet 13273a9f79d14d13b2569c8e082f7a11 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 08:57:41.556684 21084 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1e12f111846c7968bcd95be8ebc5f (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 08:57:41.557781 21088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d5ba27eb7e4d2e9aea1fff2b0ab1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.554774 21090 tablet_service.cc:1467] Processing CreateTablet for tablet 8e23fe994c2d4a818f29a1c9c4caf4c3 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 08:57:41.559029 21085 tablet_service.cc:1467] Processing CreateTablet for tablet 707dcb1fdaf64a2e9f926b3c0c746c55 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 08:57:41.560395 21087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13273a9f79d14d13b2569c8e082f7a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.561077 21090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e23fe994c2d4a818f29a1c9c4caf4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.562000 21084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d1e12f111846c7968bcd95be8ebc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.562597 21085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707dcb1fdaf64a2e9f926b3c0c746c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.558720 21214 tablet_bootstrap.cc:492] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:41.552397 21089 tablet_service.cc:1467] Processing CreateTablet for tablet 060a086354884b0786d31a6d67b07f81 (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 08:57:41.564586 21214 ts_tablet_manager.cc:1397] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.092s user 0.023s sys 0.006s
I20250114 08:57:41.564385 21083 tablet_service.cc:1467] Processing CreateTablet for tablet a69b04a506c84abda09636ce474bb7ce (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 08:57:41.568921 21082 tablet_service.cc:1467] Processing CreateTablet for tablet f559c3fc5ad7468f9b13ba6572cc25cd (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 08:57:41.572749 21081 tablet_service.cc:1467] Processing CreateTablet for tablet 11dd206a2b6246a1935e5b943ab23baf (DEFAULT_TABLE table=test-workload [id=7eb11e4892874f019d69082ca4cc9e72]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 08:57:41.574752 21089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060a086354884b0786d31a6d67b07f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.575466 21082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f559c3fc5ad7468f9b13ba6572cc25cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.576154 21083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69b04a506c84abda09636ce474bb7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.576830 21081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11dd206a2b6246a1935e5b943ab23baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:57:41.567605 21214 raft_consensus.cc:357] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.587666 21214 raft_consensus.cc:383] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.589840 21214 raft_consensus.cc:738] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.591533 21214 consensus_queue.cc:260] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.593092 21215 tablet_bootstrap.cc:654] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.607941 21215 tablet_bootstrap.cc:492] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.608587 21215 ts_tablet_manager.cc:1397] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.005s
I20250114 08:57:41.611493 21215 raft_consensus.cc:357] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.612695 21215 raft_consensus.cc:383] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.612998 21215 raft_consensus.cc:738] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.613880 21215 consensus_queue.cc:260] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.619742 21219 tablet_bootstrap.cc:492] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:41.624958 21214 ts_tablet_manager.cc:1428] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.060s user 0.008s sys 0.025s
I20250114 08:57:41.626753 21050 heartbeater.cc:502] Master 127.19.231.190:45087 was elected leader, sending a full tablet report...
I20250114 08:57:41.645758 21199 heartbeater.cc:502] Master 127.19.231.190:45087 was elected leader, sending a full tablet report...
I20250114 08:57:41.646242 21215 ts_tablet_manager.cc:1428] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.037s user 0.010s sys 0.025s
I20250114 08:57:41.647205 21215 tablet_bootstrap.cc:492] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.650619 21214 tablet_bootstrap.cc:492] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.652631 21219 tablet_bootstrap.cc:654] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.653721 21215 tablet_bootstrap.cc:654] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.658654 21214 tablet_bootstrap.cc:654] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.664567 21219 tablet_bootstrap.cc:492] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:41.665081 21219 ts_tablet_manager.cc:1397] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.007s
I20250114 08:57:41.668973 21219 raft_consensus.cc:357] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.669806 21219 raft_consensus.cc:383] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.670107 21219 raft_consensus.cc:738] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.671722 21214 tablet_bootstrap.cc:492] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:41.672219 21214 ts_tablet_manager.cc:1397] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.002s
I20250114 08:57:41.672245 21219 consensus_queue.cc:260] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.674243 21215 tablet_bootstrap.cc:492] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.674749 21215 ts_tablet_manager.cc:1397] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.011s
I20250114 08:57:41.675190 21214 raft_consensus.cc:357] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.676108 21214 raft_consensus.cc:383] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.676417 21214 raft_consensus.cc:738] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.677332 21214 consensus_queue.cc:260] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.677686 21215 raft_consensus.cc:357] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.678398 21215 raft_consensus.cc:383] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.678704 21215 raft_consensus.cc:738] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.679822 21214 ts_tablet_manager.cc:1428] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250114 08:57:41.679581 21215 consensus_queue.cc:260] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.680815 21214 tablet_bootstrap.cc:492] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.682320 21215 ts_tablet_manager.cc:1428] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250114 08:57:41.682652 21219 ts_tablet_manager.cc:1428] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20250114 08:57:41.683219 21215 tablet_bootstrap.cc:492] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.683353 21125 heartbeater.cc:502] Master 127.19.231.190:45087 was elected leader, sending a full tablet report...
I20250114 08:57:41.688637 21219 tablet_bootstrap.cc:492] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:41.693819 21214 tablet_bootstrap.cc:654] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.694970 21219 tablet_bootstrap.cc:654] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.704247 21219 tablet_bootstrap.cc:492] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:41.704844 21219 ts_tablet_manager.cc:1397] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20250114 08:57:41.707567 21219 raft_consensus.cc:357] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.708124 21219 raft_consensus.cc:383] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.708438 21219 raft_consensus.cc:738] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.710101 21219 consensus_queue.cc:260] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.711423 21214 tablet_bootstrap.cc:492] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:41.711956 21214 ts_tablet_manager.cc:1397] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.015s
I20250114 08:57:41.712365 21219 ts_tablet_manager.cc:1428] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20250114 08:57:41.713187 21219 tablet_bootstrap.cc:492] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:41.714363 21215 tablet_bootstrap.cc:654] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.714854 21214 raft_consensus.cc:357] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.715674 21214 raft_consensus.cc:383] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.715975 21214 raft_consensus.cc:738] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.716799 21214 consensus_queue.cc:260] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.718084 21219 tablet_bootstrap.cc:654] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.720299 21214 ts_tablet_manager.cc:1428] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250114 08:57:41.721727 21214 tablet_bootstrap.cc:492] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.730749 21214 tablet_bootstrap.cc:654] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.732648 21219 tablet_bootstrap.cc:492] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:41.733170 21219 ts_tablet_manager.cc:1397] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20250114 08:57:41.734900 21215 tablet_bootstrap.cc:492] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.735378 21215 ts_tablet_manager.cc:1397] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.012s
I20250114 08:57:41.735913 21219 raft_consensus.cc:357] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.736575 21219 raft_consensus.cc:383] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.736814 21219 raft_consensus.cc:738] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.737335 21219 consensus_queue.cc:260] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.737896 21215 raft_consensus.cc:357] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.738557 21215 raft_consensus.cc:383] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.738863 21215 raft_consensus.cc:738] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.739215 21219 ts_tablet_manager.cc:1428] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20250114 08:57:41.740048 21219 tablet_bootstrap.cc:492] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:41.739602 21215 consensus_queue.cc:260] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.742274 21215 ts_tablet_manager.cc:1428] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250114 08:57:41.743279 21215 tablet_bootstrap.cc:492] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.744453 21214 tablet_bootstrap.cc:492] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:41.745260 21214 ts_tablet_manager.cc:1397] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20250114 08:57:41.746943 21219 tablet_bootstrap.cc:654] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.748378 21214 raft_consensus.cc:357] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.749279 21214 raft_consensus.cc:383] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.749650 21215 tablet_bootstrap.cc:654] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.749665 21214 raft_consensus.cc:738] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.750798 21214 consensus_queue.cc:260] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.757025 21214 ts_tablet_manager.cc:1428] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20250114 08:57:41.758603 21214 tablet_bootstrap.cc:492] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.767743 21214 tablet_bootstrap.cc:654] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.770949 21215 tablet_bootstrap.cc:492] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.771507 21215 ts_tablet_manager.cc:1397] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.002s
I20250114 08:57:41.774976 21215 raft_consensus.cc:357] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.775789 21215 raft_consensus.cc:383] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.776083 21215 raft_consensus.cc:738] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.776816 21215 consensus_queue.cc:260] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.777241 21219 tablet_bootstrap.cc:492] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:41.778020 21219 ts_tablet_manager.cc:1397] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.020s
I20250114 08:57:41.779279 21215 ts_tablet_manager.cc:1428] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 08:57:41.780875 21219 raft_consensus.cc:357] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.781652 21219 raft_consensus.cc:383] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.782043 21219 raft_consensus.cc:738] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.782948 21219 consensus_queue.cc:260] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.785516 21219 ts_tablet_manager.cc:1428] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:57:41.786257 21219 tablet_bootstrap.cc:492] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:41.786739 21220 raft_consensus.cc:491] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.787209 21220 raft_consensus.cc:513] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.789079 21214 tablet_bootstrap.cc:492] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:41.789561 21214 ts_tablet_manager.cc:1397] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.006s
I20250114 08:57:41.789700 21220 leader_election.cc:290] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:41.792488 21214 raft_consensus.cc:357] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.793092 21219 tablet_bootstrap.cc:654] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.793303 21214 raft_consensus.cc:383] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.793632 21214 raft_consensus.cc:738] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.794384 21214 consensus_queue.cc:260] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.786290 21215 tablet_bootstrap.cc:492] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.796888 21214 ts_tablet_manager.cc:1428] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20250114 08:57:41.797770 21214 tablet_bootstrap.cc:492] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.802306 21215 tablet_bootstrap.cc:654] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.802764 21224 raft_consensus.cc:491] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.803260 21224 raft_consensus.cc:513] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.804456 21214 tablet_bootstrap.cc:654] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.810770 21217 raft_consensus.cc:491] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.811283 21217 raft_consensus.cc:513] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.817005 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:41.817905 21025 raft_consensus.cc:2463] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 0.
I20250114 08:57:41.805868 21224 leader_election.cc:290] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.823653 21217 leader_election.cc:290] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.824323 21217 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.825996 21139 leader_election.cc:304] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:41.829432 21220 raft_consensus.cc:2798] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:41.830050 21220 raft_consensus.cc:491] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:41.830533 21220 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.830192 21217 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.832667 21224 raft_consensus.cc:491] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.833355 21224 raft_consensus.cc:513] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.832258 21217 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.835995 21224 leader_election.cc:290] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.844152 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
W20250114 08:57:41.846189 21139 leader_election.cc:343] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:41.852550 21215 tablet_bootstrap.cc:492] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.853538 21215 ts_tablet_manager.cc:1397] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.069s user 0.023s sys 0.012s
I20250114 08:57:41.857662 21215 raft_consensus.cc:357] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.858758 21215 raft_consensus.cc:383] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.859153 21215 raft_consensus.cc:738] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.860133 21215 consensus_queue.cc:260] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.872503 21215 ts_tablet_manager.cc:1428] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20250114 08:57:41.873870 21215 tablet_bootstrap.cc:492] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.889209 21215 tablet_bootstrap.cc:654] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.892704 21224 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.893513 21224 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.896171 21224 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.908814 21215 tablet_bootstrap.cc:492] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.909549 21215 ts_tablet_manager.cc:1397] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.001s
W20250114 08:57:41.913080 20971 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:57:41.912971 21215 raft_consensus.cc:357] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.913905 21215 raft_consensus.cc:383] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.914299 21215 raft_consensus.cc:738] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.903928 21220 raft_consensus.cc:513] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.919663 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:41.921125 21174 raft_consensus.cc:2463] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 0.
I20250114 08:57:41.920253 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:41.923451 20990 leader_election.cc:304] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:41.924499 21238 raft_consensus.cc:2798] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:41.924921 21238 raft_consensus.cc:491] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:41.925459 21238 raft_consensus.cc:3054] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.926460 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
W20250114 08:57:41.924499 20990 leader_election.cc:343] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:41.927340 21220 leader_election.cc:290] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:41.927851 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
I20250114 08:57:41.928588 21099 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 0.
I20250114 08:57:41.932592 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:41.932660 21172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:41.924943 21215 consensus_queue.cc:260] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.933658 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:41.934340 21174 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
W20250114 08:57:41.934434 21065 leader_election.cc:343] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:41.935408 21215 ts_tablet_manager.cc:1428] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250114 08:57:41.936443 21215 tablet_bootstrap.cc:492] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:41.936928 21065 leader_election.cc:304] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:41.937322 21214 tablet_bootstrap.cc:492] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:41.937958 21214 ts_tablet_manager.cc:1397] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.140s user 0.054s sys 0.068s
I20250114 08:57:41.938134 21238 raft_consensus.cc:513] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.938660 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:41.940430 21238 leader_election.cc:290] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.941180 21224 raft_consensus.cc:2798] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:41.941627 21224 raft_consensus.cc:491] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:41.942019 21224 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.942767 21219 tablet_bootstrap.cc:492] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:41.943290 21219 ts_tablet_manager.cc:1397] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.157s user 0.024s sys 0.045s
I20250114 08:57:41.941298 21214 raft_consensus.cc:357] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.943874 21214 raft_consensus.cc:383] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:41.944245 21214 raft_consensus.cc:738] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.945016 21214 consensus_queue.cc:260] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.946151 21219 raft_consensus.cc:357] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.946924 21219 raft_consensus.cc:383] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250114 08:57:41.947614 21139 leader_election.cc:343] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:41.947767 21214 ts_tablet_manager.cc:1428] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 08:57:41.947222 21219 raft_consensus.cc:738] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:41.948246 21173 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:41.948853 21219 consensus_queue.cc:260] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.949255 21224 raft_consensus.cc:513] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.938213 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:41.950908 21025 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.952728 21214 tablet_bootstrap.cc:492] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:41.953194 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:41.954099 21024 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:41.954598 21215 tablet_bootstrap.cc:654] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.955313 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:41.956120 21023 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:41.957176 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:41.957832 21173 raft_consensus.cc:3054] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.957520 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:41.959713 21022 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:41.961041 21224 leader_election.cc:290] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:41.950893 21065 leader_election.cc:304] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:41.964433 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:41.964843 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:41.965248 21174 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.965474 21022 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.968263 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:41.970813 21065 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters: 8905c307b2d04181a6333a806893cbdc
W20250114 08:57:41.971680 20990 leader_election.cc:343] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:41.972667 21224 raft_consensus.cc:2798] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:41.975733 21224 raft_consensus.cc:491] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:41.974872 21022 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:41.978408 21214 tablet_bootstrap.cc:654] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:41.976220 21224 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.985507 21220 raft_consensus.cc:491] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:41.966471 21173 raft_consensus.cc:2463] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 1.
I20250114 08:57:41.985978 21220 raft_consensus.cc:513] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:41.987107 21065 leader_election.cc:304] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:41.987849 21025 raft_consensus.cc:2463] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 1.
I20250114 08:57:41.988262 21220 leader_election.cc:290] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:41.985248 21219 ts_tablet_manager.cc:1428] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.042s user 0.016s sys 0.004s
I20250114 08:57:41.983590 21174 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:41.991719 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:41.976015 20990 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters: 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:41.994215 21238 raft_consensus.cc:2798] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:41.996985 21238 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:41.997356 21238 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:41.997253 21139 leader_election.cc:304] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters: 08a5ac200b51404694fa174ebc6fe0d5
W20250114 08:57:41.995235 20990 leader_election.cc:343] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:41.998775 20990 leader_election.cc:304] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters: 08a5ac200b51404694fa174ebc6fe0d5
I20250114 08:57:41.993580 21219 tablet_bootstrap.cc:492] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:42.000437 21217 raft_consensus.cc:2798] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.000856 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
I20250114 08:57:42.004357 21220 raft_consensus.cc:2798] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.004917 21220 raft_consensus.cc:695] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:42.005750 21220 consensus_queue.cc:237] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.007428 21099 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 0.
I20250114 08:57:42.007714 21238 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.009052 21139 leader_election.cc:304] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:42.010041 21242 raft_consensus.cc:2798] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:42.010428 21242 raft_consensus.cc:491] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:42.010776 21242 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.010926 21219 tablet_bootstrap.cc:654] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.015738 21215 tablet_bootstrap.cc:492] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:41.992689 21022 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 0.
I20250114 08:57:42.017659 21215 ts_tablet_manager.cc:1397] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.011s
I20250114 08:57:42.017385 21217 raft_consensus.cc:695] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:42.018810 21238 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.018690 21217 consensus_queue.cc:237] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.019944 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.020659 21215 raft_consensus.cc:357] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
W20250114 08:57:42.021205 20990 leader_election.cc:343] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:42.021413 21215 raft_consensus.cc:383] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.022221 21245 raft_consensus.cc:2798] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:42.025208 21245 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:42.025605 21245 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.024729 21215 raft_consensus.cc:738] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.027885 21248 raft_consensus.cc:2798] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.027747 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:42.034740 21215 consensus_queue.cc:260] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.035534 21214 tablet_bootstrap.cc:492] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:42.036024 21214 ts_tablet_manager.cc:1397] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.084s user 0.014s sys 0.020s
I20250114 08:57:42.037353 21215 ts_tablet_manager.cc:1428] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20250114 08:57:42.038311 21215 tablet_bootstrap.cc:492] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:42.034346 21242 raft_consensus.cc:513] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.038856 21214 raft_consensus.cc:357] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.039722 21214 raft_consensus.cc:383] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.040036 21214 raft_consensus.cc:738] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.040292 21242 leader_election.cc:290] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:42.041122 21214 consensus_queue.cc:260] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.042392 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:42.038928 20919 catalog_manager.cc:5526] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 1 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:42.045013 21100 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.045266 21215 tablet_bootstrap.cc:654] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.043440 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:42.045689 21224 raft_consensus.cc:513] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.048245 21022 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.043905 20920 catalog_manager.cc:5526] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.049957 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:42.050302 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.050657 21025 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.050868 21174 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.051581 21219 tablet_bootstrap.cc:492] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:42.052107 21219 ts_tablet_manager.cc:1397] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.003s
I20250114 08:57:42.055279 21219 raft_consensus.cc:357] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.056030 21219 raft_consensus.cc:383] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.056355 21219 raft_consensus.cc:738] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.059708 21214 ts_tablet_manager.cc:1428] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20250114 08:57:42.059070 21219 consensus_queue.cc:260] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.056764 21224 leader_election.cc:290] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.060736 21214 tablet_bootstrap.cc:492] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:42.064491 21022 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 1.
I20250114 08:57:42.064960 21174 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.066712 21100 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 1.
I20250114 08:57:42.068387 21248 raft_consensus.cc:695] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:42.067382 21025 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.069453 21139 leader_election.cc:304] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:42.069828 21065 leader_election.cc:304] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:42.070567 21220 raft_consensus.cc:2798] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.071066 21224 raft_consensus.cc:2798] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.071316 21220 raft_consensus.cc:695] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:42.071630 21224 raft_consensus.cc:695] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:42.072610 21224 consensus_queue.cc:237] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.072757 21220 consensus_queue.cc:237] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.067454 21214 tablet_bootstrap.cc:654] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.076885 21248 consensus_queue.cc:237] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.077849 21242 raft_consensus.cc:491] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.082178 21219 ts_tablet_manager.cc:1428] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.030s user 0.014s sys 0.010s
I20250114 08:57:42.082453 21242 raft_consensus.cc:513] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.083370 21219 tablet_bootstrap.cc:492] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:42.104552 21258 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.093319 21245 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.107501 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559c3fc5ad7468f9b13ba6572cc25cd" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:42.107659 21242 leader_election.cc:290] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:42.111119 21099 raft_consensus.cc:2388] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26d40bfe96134eb0b1478d2664b2086e in current term 1: Already voted for candidate 08a5ac200b51404694fa174ebc6fe0d5 in this term.
I20250114 08:57:42.111372 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559c3fc5ad7468f9b13ba6572cc25cd" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
I20250114 08:57:42.119676 20990 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e; no voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:42.121059 21245 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.121402 21217 raft_consensus.cc:2743] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 08:57:42.123158 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.123046 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
W20250114 08:57:42.116734 21139 leader_election.cc:343] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:42.124045 21025 raft_consensus.cc:2388] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 08a5ac200b51404694fa174ebc6fe0d5 in current term 1: Already voted for candidate 26d40bfe96134eb0b1478d2664b2086e in this term.
W20250114 08:57:42.124523 21139 leader_election.cc:343] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:57:42.125043 21139 leader_election.cc:304] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8905c307b2d04181a6333a806893cbdc; no voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.125917 21242 raft_consensus.cc:2743] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:42.112125 21258 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.129688 21174 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.135715 21214 tablet_bootstrap.cc:492] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:42.136453 21214 ts_tablet_manager.cc:1397] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.076s user 0.030s sys 0.002s
I20250114 08:57:42.140113 21215 tablet_bootstrap.cc:492] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:42.144917 21215 ts_tablet_manager.cc:1397] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.107s user 0.021s sys 0.012s
I20250114 08:57:42.147159 21258 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:42.147862 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:42.148730 21025 raft_consensus.cc:2388] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8905c307b2d04181a6333a806893cbdc in current term 1: Already voted for candidate 26d40bfe96134eb0b1478d2664b2086e in this term.
I20250114 08:57:42.149006 21214 raft_consensus.cc:357] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.149760 21214 raft_consensus.cc:383] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.150065 21214 raft_consensus.cc:738] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.150576 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
I20250114 08:57:42.151350 21099 raft_consensus.cc:2388] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8905c307b2d04181a6333a806893cbdc in current term 1: Already voted for candidate 08a5ac200b51404694fa174ebc6fe0d5 in this term.
I20250114 08:57:42.150915 21214 consensus_queue.cc:260] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.153108 21139 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8905c307b2d04181a6333a806893cbdc; no voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.154222 21214 ts_tablet_manager.cc:1428] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250114 08:57:42.155436 21174 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.157016 21214 tablet_bootstrap.cc:492] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:42.159157 21065 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters: 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.160563 21245 raft_consensus.cc:2798] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.161182 21245 raft_consensus.cc:695] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:42.154462 21219 tablet_bootstrap.cc:654] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.162037 21245 consensus_queue.cc:237] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.174060 21215 raft_consensus.cc:357] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.174919 21215 raft_consensus.cc:383] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.175204 21215 raft_consensus.cc:738] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.175834 21215 consensus_queue.cc:260] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.177065 21242 raft_consensus.cc:2743] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:42.178107 21215 ts_tablet_manager.cc:1428] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.033s user 0.001s sys 0.005s
I20250114 08:57:42.179102 21215 tablet_bootstrap.cc:492] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:42.181488 21219 tablet_bootstrap.cc:492] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:42.181933 21219 ts_tablet_manager.cc:1397] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.000s
I20250114 08:57:42.183939 21219 raft_consensus.cc:357] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.184551 21219 raft_consensus.cc:383] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.184883 21215 tablet_bootstrap.cc:654] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.184792 20919 catalog_manager.cc:5526] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 1 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:42.186285 21214 tablet_bootstrap.cc:654] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.184789 21219 raft_consensus.cc:738] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.191120 21214 tablet_bootstrap.cc:492] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:42.190953 21219 consensus_queue.cc:260] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.191605 21214 ts_tablet_manager.cc:1397] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.005s
I20250114 08:57:42.193477 21215 tablet_bootstrap.cc:492] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:42.193632 21219 ts_tablet_manager.cc:1428] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250114 08:57:42.193915 21215 ts_tablet_manager.cc:1397] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250114 08:57:42.194854 21219 tablet_bootstrap.cc:492] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:42.194537 21214 raft_consensus.cc:357] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.195363 21214 raft_consensus.cc:383] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.195689 21214 raft_consensus.cc:738] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.196373 21214 consensus_queue.cc:260] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.196370 21215 raft_consensus.cc:357] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.197364 21215 raft_consensus.cc:383] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.197654 21215 raft_consensus.cc:738] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.198259 21215 consensus_queue.cc:260] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.199028 21214 ts_tablet_manager.cc:1428] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:57:42.200314 21215 ts_tablet_manager.cc:1428] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:57:42.201153 21215 tablet_bootstrap.cc:492] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc: Bootstrap starting.
I20250114 08:57:42.204558 21214 tablet_bootstrap.cc:492] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e: Bootstrap starting.
I20250114 08:57:42.206073 21219 tablet_bootstrap.cc:654] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.206986 21215 tablet_bootstrap.cc:654] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.211239 21214 tablet_bootstrap.cc:654] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.212978 20920 catalog_manager.cc:5526] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 1 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.218678 21219 tablet_bootstrap.cc:492] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:42.219168 21219 ts_tablet_manager.cc:1397] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.007s
I20250114 08:57:42.221272 21214 tablet_bootstrap.cc:492] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e: No bootstrap required, opened a new log
I20250114 08:57:42.221773 21214 ts_tablet_manager.cc:1397] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20250114 08:57:42.221858 21219 raft_consensus.cc:357] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.222600 21219 raft_consensus.cc:383] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.222877 21219 raft_consensus.cc:738] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.223603 21219 consensus_queue.cc:260] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.224887 21214 raft_consensus.cc:357] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.225623 21214 raft_consensus.cc:383] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.225929 21214 raft_consensus.cc:738] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.226655 21214 consensus_queue.cc:260] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.228919 21214 ts_tablet_manager.cc:1428] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:57:42.229228 21215 tablet_bootstrap.cc:492] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc: No bootstrap required, opened a new log
I20250114 08:57:42.226053 21219 ts_tablet_manager.cc:1428] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250114 08:57:42.229655 21215 ts_tablet_manager.cc:1397] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.013s
I20250114 08:57:42.232681 21219 tablet_bootstrap.cc:492] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:42.232254 21215 raft_consensus.cc:357] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.232983 21215 raft_consensus.cc:383] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.233266 21215 raft_consensus.cc:738] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.233923 21215 consensus_queue.cc:260] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.236052 21215 ts_tablet_manager.cc:1428] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:57:42.238373 21219 tablet_bootstrap.cc:654] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.240036 20922 catalog_manager.cc:5526] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 1 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.241977 20922 catalog_manager.cc:5526] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 1 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.243606 21219 tablet_bootstrap.cc:492] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:42.244055 21219 ts_tablet_manager.cc:1397] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20250114 08:57:42.246222 21219 raft_consensus.cc:357] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.246686 21219 raft_consensus.cc:383] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.246903 21219 raft_consensus.cc:738] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.247949 21219 consensus_queue.cc:260] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.250000 21219 ts_tablet_manager.cc:1428] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 08:57:42.250824 21219 tablet_bootstrap.cc:492] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5: Bootstrap starting.
I20250114 08:57:42.255204 21219 tablet_bootstrap.cc:654] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5: Neither blocks nor log segments found. Creating new log.
I20250114 08:57:42.261112 21219 tablet_bootstrap.cc:492] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5: No bootstrap required, opened a new log
I20250114 08:57:42.261557 21219 ts_tablet_manager.cc:1397] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20250114 08:57:42.263465 21219 raft_consensus.cc:357] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.263959 21219 raft_consensus.cc:383] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:57:42.264153 21219 raft_consensus.cc:738] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Initialized, Role: FOLLOWER
I20250114 08:57:42.264653 21219 consensus_queue.cc:260] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.266027 21219 ts_tablet_manager.cc:1428] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250114 08:57:42.281859 21217 raft_consensus.cc:491] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.282323 21217 raft_consensus.cc:513] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.283957 21217 leader_election.cc:290] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:42.284914 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:42.285043 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
I20250114 08:57:42.285482 21174 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 0.
I20250114 08:57:42.285657 21099 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 0.
I20250114 08:57:42.286429 20990 leader_election.cc:304] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:42.287144 21217 raft_consensus.cc:2798] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:42.287424 21217 raft_consensus.cc:491] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:42.287740 21217 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.292452 21217 raft_consensus.cc:513] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.293962 21217 leader_election.cc:290] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Requested vote from peers 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:42.294901 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.295110 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:42.295382 21174 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.295632 21099 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.300361 21099 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 1.
I20250114 08:57:42.301157 21174 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 1.
I20250114 08:57:42.301331 20990 leader_election.cc:304] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:42.302083 21217 raft_consensus.cc:2798] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.302524 21217 raft_consensus.cc:695] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:42.303236 21217 consensus_queue.cc:237] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:42.310655 20919 catalog_manager.cc:5526] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.356555 21248 raft_consensus.cc:491] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.357046 21248 raft_consensus.cc:513] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.358611 21248 leader_election.cc:290] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.359597 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:42.359771 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:42.360206 21025 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:42.360276 21174 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:42.361214 21065 leader_election.cc:304] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:42.361853 21248 raft_consensus.cc:2798] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:42.362128 21248 raft_consensus.cc:491] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:42.362373 21248 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.366703 21245 raft_consensus.cc:491] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.367095 21248 raft_consensus.cc:513] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.367116 21245 raft_consensus.cc:513] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.368757 21248 leader_election.cc:290] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.368875 21245 leader_election.cc:290] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.369727 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:42.370028 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559c3fc5ad7468f9b13ba6572cc25cd" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:42.370422 21025 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.370056 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.370494 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559c3fc5ad7468f9b13ba6572cc25cd" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:42.370837 21022 raft_consensus.cc:2463] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:42.371102 21174 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.371178 21173 raft_consensus.cc:2463] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 0.
I20250114 08:57:42.372166 21065 leader_election.cc:304] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:42.373589 21245 raft_consensus.cc:2798] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:42.373970 21245 raft_consensus.cc:491] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:42.374313 21245 raft_consensus.cc:3054] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.377517 21025 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.378688 21065 leader_election.cc:304] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:42.378815 21174 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.379030 21245 raft_consensus.cc:513] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.379539 21248 raft_consensus.cc:2798] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.380092 21248 raft_consensus.cc:695] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:42.381130 21245 leader_election.cc:290] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.380935 21248 consensus_queue.cc:237] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.381943 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559c3fc5ad7468f9b13ba6572cc25cd" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:42.382136 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559c3fc5ad7468f9b13ba6572cc25cd" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.382534 21025 raft_consensus.cc:3054] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.382714 21174 raft_consensus.cc:3054] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.388553 21174 raft_consensus.cc:2463] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.389456 21065 leader_election.cc:304] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:42.389971 21025 raft_consensus.cc:2463] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 1.
I20250114 08:57:42.389904 20919 catalog_manager.cc:5526] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 1 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.390884 21248 raft_consensus.cc:2798] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.391500 21248 raft_consensus.cc:695] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:42.392321 21248 consensus_queue.cc:237] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.402807 20919 catalog_manager.cc:5526] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 1 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.427897 21248 raft_consensus.cc:491] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.428360 21248 raft_consensus.cc:513] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.429966 21248 leader_election.cc:290] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.430940 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d40bfe96134eb0b1478d2664b2086e" is_pre_election: true
I20250114 08:57:42.431102 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:42.431604 21025 raft_consensus.cc:2388] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08a5ac200b51404694fa174ebc6fe0d5 in current term 1: Already voted for candidate 8905c307b2d04181a6333a806893cbdc in this term.
I20250114 08:57:42.432633 21065 leader_election.cc:304] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5; no voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:42.433310 21248 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.438212 21248 raft_consensus.cc:2743] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:57:42.438856 21099 raft_consensus.cc:3054] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.444885 21217 consensus_queue.cc:1035] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.456430 21249 consensus_queue.cc:1035] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:42.490476 21248 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.493441 21220 consensus_queue.cc:1035] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.502954 21248 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 08:57:42.508378 21258 consensus_queue.cc:1035] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:42.512650 21258 consensus_queue.cc:1035] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.534056 21220 consensus_queue.cc:1035] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.570809 21224 consensus_queue.cc:1035] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.580927 21245 consensus_queue.cc:1035] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.581895 21224 consensus_queue.cc:1035] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.594260 21249 raft_consensus.cc:491] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:57:42.594803 21249 raft_consensus.cc:513] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.597375 21249 leader_election.cc:290] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.599087 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5" is_pre_election: true
I20250114 08:57:42.599114 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc" is_pre_election: true
I20250114 08:57:42.599907 21099 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 0.
I20250114 08:57:42.599819 21224 consensus_queue.cc:1035] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:57:42.599979 21173 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 0.
I20250114 08:57:42.601383 20990 leader_election.cc:304] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:42.603385 21249 raft_consensus.cc:2798] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:57:42.603824 21249 raft_consensus.cc:491] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:57:42.604233 21249 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.613654 21249 raft_consensus.cc:513] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.615947 21249 leader_election.cc:290] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:42.617216 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:42.617547 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:42.617909 21174 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.618160 21099 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:57:42.623104 21174 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 1.
I20250114 08:57:42.624322 21099 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 1.
I20250114 08:57:42.624382 20990 leader_election.cc:304] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:42.625283 21249 raft_consensus.cc:2798] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:57:42.625785 21249 raft_consensus.cc:695] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:42.626617 21249 consensus_queue.cc:237] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:42.634835 20920 catalog_manager.cc:5526] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:42.740658 21249 consensus_queue.cc:1035] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.750497 21217 consensus_queue.cc:1035] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.846910 21224 consensus_queue.cc:1035] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.857381 21248 consensus_queue.cc:1035] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.874070 21248 consensus_queue.cc:1035] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.879390 21248 consensus_queue.cc:1035] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:42.882091 21173 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "060a086354884b0786d31a6d67b07f81"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58560
I20250114 08:57:42.882856 21173 raft_consensus.cc:604] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.903060 21099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:56016
I20250114 08:57:42.903715 21099 raft_consensus.cc:604] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.909494 21099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:56016
I20250114 08:57:42.909965 21099 raft_consensus.cc:604] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.914337 21099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:56016
I20250114 08:57:42.914830 21099 raft_consensus.cc:604] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:42.928426 21022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:58156
I20250114 08:57:42.928987 21022 raft_consensus.cc:604] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Received request to transfer leadership to TS 08a5ac200b51404694fa174ebc6fe0d5
I20250114 08:57:42.936064 21174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
mode: GRACEFUL
new_leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:58560
I20250114 08:57:42.936604 21174 raft_consensus.cc:604] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Received request to transfer leadership to TS 08a5ac200b51404694fa174ebc6fe0d5
I20250114 08:57:42.943181 21022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58156
I20250114 08:57:42.943738 21022 raft_consensus.cc:604] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Received request to transfer leadership to TS 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:43.015601 21238 raft_consensus.cc:988] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:57:43.016022 21217 raft_consensus.cc:1076] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Signalling peer 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:57:43.017467 21174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58534
I20250114 08:57:43.018182 21174 raft_consensus.cc:491] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.018569 21174 raft_consensus.cc:3054] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.024111 21174 raft_consensus.cc:513] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.025660 21174 leader_election.cc:290] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 2 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:43.026508 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:43.026747 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dd206a2b6246a1935e5b943ab23baf" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:43.027160 21022 raft_consensus.cc:3049] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.027235 21099 raft_consensus.cc:3054] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.027576 21022 raft_consensus.cc:738] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:43.028366 21022 consensus_queue.cc:260] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.029738 21022 raft_consensus.cc:3054] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.032902 21099 raft_consensus.cc:2463] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 2.
I20250114 08:57:43.034279 21139 leader_election.cc:304] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:43.034739 21258 raft_consensus.cc:988] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: : Instructing follower 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:57:43.035230 21220 raft_consensus.cc:1076] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Signalling peer 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:57:43.035169 21242 raft_consensus.cc:2798] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.036232 21242 raft_consensus.cc:695] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:43.036878 21099 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:56002
I20250114 08:57:43.037047 21242 consensus_queue.cc:237] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.037564 21099 raft_consensus.cc:491] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.037927 21099 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.036239 21022 raft_consensus.cc:2463] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 2.
I20250114 08:57:43.042560 21248 raft_consensus.cc:988] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.043011 21224 raft_consensus.cc:1076] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.044186 21099 raft_consensus.cc:513] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.044631 21022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58144
I20250114 08:57:43.045224 21022 raft_consensus.cc:491] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.045611 21022 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.045871 20919 catalog_manager.cc:5526] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 1 to 2, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 2 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:43.046619 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:43.047061 21099 leader_election.cc:290] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 2 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:43.047232 21025 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.048622 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:43.049216 21174 raft_consensus.cc:3049] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.049536 21174 raft_consensus.cc:738] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:43.050370 21174 consensus_queue.cc:260] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.051647 21174 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.052582 21022 raft_consensus.cc:513] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.054603 21022 leader_election.cc:290] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:43.055866 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:43.056241 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:43.056542 21099 raft_consensus.cc:3049] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.056881 21173 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.056888 21099 raft_consensus.cc:738] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:43.057921 21099 consensus_queue.cc:260] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.059199 21099 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.059638 21025 raft_consensus.cc:2463] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 2.
I20250114 08:57:43.060705 21065 leader_election.cc:304] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:43.061873 21224 raft_consensus.cc:2798] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.062347 21224 raft_consensus.cc:695] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:43.063160 21224 consensus_queue.cc:237] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.063832 21174 raft_consensus.cc:2463] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 2.
I20250114 08:57:43.066815 21099 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.066815 21173 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.068049 20990 leader_election.cc:304] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:43.068863 21217 raft_consensus.cc:2798] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.069353 21217 raft_consensus.cc:695] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:43.069578 21238 consensus_queue.cc:1035] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:57:43.070168 21217 consensus_queue.cc:237] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.072882 20919 catalog_manager.cc:5526] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 1 to 2, leader changed from 8905c307b2d04181a6333a806893cbdc (127.19.231.131) to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 2 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:43.079578 20920 catalog_manager.cc:5526] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 1 to 2, leader changed from 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:43.085461 21217 consensus_queue.cc:1035] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250114 08:57:43.097326 21217 raft_consensus.cc:988] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:57:43.097803 21238 raft_consensus.cc:1076] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Signalling peer 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:57:43.099282 21099 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:56006
I20250114 08:57:43.099915 21099 raft_consensus.cc:491] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.100302 21099 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.105767 21099 raft_consensus.cc:513] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.107241 21099 leader_election.cc:290] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 2 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:43.108201 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:43.108799 21025 raft_consensus.cc:3049] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.108824 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:43.109232 21025 raft_consensus.cc:738] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:43.109503 21174 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.110091 21025 consensus_queue.cc:260] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.111430 21025 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.116603 21174 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 2.
I20250114 08:57:43.117866 21065 leader_election.cc:304] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:43.118017 21025 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 2.
I20250114 08:57:43.118553 21224 raft_consensus.cc:2798] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.119026 21224 raft_consensus.cc:695] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:43.119760 21224 consensus_queue.cc:237] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.125751 20919 catalog_manager.cc:5526] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 1 to 2, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 2 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:43.130414 21220 raft_consensus.cc:988] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.130894 21242 raft_consensus.cc:1076] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.134925 21025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "060a086354884b0786d31a6d67b07f81"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58142
I20250114 08:57:43.135459 21025 raft_consensus.cc:491] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.135751 21025 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.140877 21025 raft_consensus.cc:513] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.142362 21025 leader_election.cc:290] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:43.143224 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:43.143410 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:43.143806 21099 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.143925 21174 raft_consensus.cc:3049] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.144253 21174 raft_consensus.cc:738] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:43.144884 21174 consensus_queue.cc:260] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.145817 21174 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.149113 21099 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.150262 20990 leader_election.cc:304] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:43.151041 21238 raft_consensus.cc:2798] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.151434 21238 raft_consensus.cc:695] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:43.151504 21174 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.152256 21238 consensus_queue.cc:237] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.158430 20919 catalog_manager.cc:5526] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 1 to 2, leader changed from 8905c307b2d04181a6333a806893cbdc (127.19.231.131) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:43.197690 21248 raft_consensus.cc:988] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.198109 21248 raft_consensus.cc:1076] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.199393 21025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58144
I20250114 08:57:43.199889 21025 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.200204 21025 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.206688 21025 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.208544 21025 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:43.209597 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:43.209730 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:43.210081 21099 raft_consensus.cc:3049] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.210377 21174 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.210422 21099 raft_consensus.cc:738] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:43.211390 21099 consensus_queue.cc:260] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.212801 21099 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.217563 21174 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.218693 20990 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:43.219415 21249 raft_consensus.cc:2798] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.219362 21099 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.219992 21249 raft_consensus.cc:695] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:43.220769 21249 consensus_queue.cc:237] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.226495 20919 catalog_manager.cc:5526] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 1 to 2, leader changed from 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:43.285171 21248 raft_consensus.cc:988] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.285678 21245 raft_consensus.cc:1076] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:43.287151 21025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58144
I20250114 08:57:43.287627 21025 raft_consensus.cc:491] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:43.287905 21025 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.292549 21025 raft_consensus.cc:513] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.294114 21025 leader_election.cc:290] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:43.294920 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:43.295152 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:43.295576 21099 raft_consensus.cc:3049] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:43.295683 21174 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.295960 21099 raft_consensus.cc:738] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:43.296777 21099 consensus_queue.cc:260] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.298017 21099 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:43.302563 21174 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.303752 20990 leader_election.cc:304] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:43.304577 21249 raft_consensus.cc:2798] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:43.305097 21249 raft_consensus.cc:695] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:43.304563 21099 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:43.305889 21249 consensus_queue.cc:237] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:43.312122 20919 catalog_manager.cc:5526] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 1 to 2, leader changed from 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:43.507104 21174 raft_consensus.cc:1270] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.508510 21245 consensus_queue.cc:1035] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.510928 21099 raft_consensus.cc:1270] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.512428 21242 consensus_queue.cc:1035] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.522778 21025 raft_consensus.cc:1270] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.524698 21245 consensus_queue.cc:1035] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:57:43.533246 21025 raft_consensus.cc:1270] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.534737 21220 consensus_queue.cc:1035] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.599918 21174 raft_consensus.cc:1270] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.601058 21245 consensus_queue.cc:1035] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.610205 21025 raft_consensus.cc:1270] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.611719 21245 consensus_queue.cc:1035] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.626103 21100 raft_consensus.cc:1270] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.627147 21249 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.633703 21174 raft_consensus.cc:1270] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.636224 21249 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.698563 21174 raft_consensus.cc:1270] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.699748 21217 consensus_queue.cc:1035] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.710409 21100 raft_consensus.cc:1270] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.711953 21217 consensus_queue.cc:1035] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.784240 21100 raft_consensus.cc:1270] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.784649 21099 raft_consensus.cc:1270] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.785418 21217 consensus_queue.cc:1035] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.785827 21249 consensus_queue.cc:1035] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.796783 21173 raft_consensus.cc:1270] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.798498 21174 raft_consensus.cc:1270] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:43.800340 21238 consensus_queue.cc:1035] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:43.801494 21314 consensus_queue.cc:1035] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:45.975531 20382 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20250114 08:57:45.976123 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:45.997560 21025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58166
I20250114 08:57:45.998095 21025 raft_consensus.cc:604] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Received request to transfer leadership to TS 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:46.000104 21025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58166
I20250114 08:57:46.000628 21025 raft_consensus.cc:604] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Received request to transfer leadership to TS 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:46.001506 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250114 08:57:46.001914 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:46.225834 21314 raft_consensus.cc:988] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:57:46.226287 21317 raft_consensus.cc:1076] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Signalling peer 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:57:46.227785 21174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58534
I20250114 08:57:46.228313 21174 raft_consensus.cc:491] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:46.228679 21174 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:46.233745 21174 raft_consensus.cc:513] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:46.235042 21174 leader_election.cc:290] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 3 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:46.235929 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:46.236099 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:46.236631 21025 raft_consensus.cc:3049] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:46.236725 21100 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:46.237043 21025 raft_consensus.cc:738] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:46.237820 21025 consensus_queue.cc:260] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:46.239053 21025 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:46.243472 21100 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 3.
I20250114 08:57:46.244726 21139 leader_election.cc:304] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:46.245527 21220 raft_consensus.cc:2798] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:46.245600 21025 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 3.
I20250114 08:57:46.246727 21220 raft_consensus.cc:695] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:46.247598 21220 consensus_queue.cc:237] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:46.254101 20922 catalog_manager.cc:5526] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 2 to 3, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 3 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250114 08:57:46.374917 21314 raft_consensus.cc:988] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:57:46.375326 21317 raft_consensus.cc:1076] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Signalling peer 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:57:46.376758 21174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58534
I20250114 08:57:46.377171 21174 raft_consensus.cc:491] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:46.377393 21174 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:46.381531 21174 raft_consensus.cc:513] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:46.383001 21174 leader_election.cc:290] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 2 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:46.383940 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:57:46.384099 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:46.384529 21025 raft_consensus.cc:3049] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:46.384641 21100 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:46.384825 21025 raft_consensus.cc:738] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:46.385571 21025 consensus_queue.cc:260] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:46.386497 21025 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:46.389688 21100 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 2.
I20250114 08:57:46.390692 21139 leader_election.cc:304] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:46.391124 21025 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 2.
I20250114 08:57:46.391577 21220 raft_consensus.cc:2798] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:46.392006 21220 raft_consensus.cc:695] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:46.392805 21220 consensus_queue.cc:237] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:46.401154 20922 catalog_manager.cc:5526] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 1 to 2, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 2 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:46.709962 21100 raft_consensus.cc:1270] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:46.711264 21220 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:46.720710 21025 raft_consensus.cc:1270] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:46.722229 21320 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:46.858852 21025 raft_consensus.cc:1270] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:46.859968 21220 consensus_queue.cc:1035] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:46.868595 21100 raft_consensus.cc:1270] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:46.870193 21220 consensus_queue.cc:1035] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:47.002774 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:47.008231 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:47.008706 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:48.009683 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:48.016116 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:48.016647 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:49.017493 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:49.023110 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:49.023577 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:50.024642 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:50.032166 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:50.032683 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:51.463210 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:51.469004 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:51.469395 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250114 08:57:51.550266 21046 debug-util.cc:398] Leaking SignalData structure 0x7b08001d44e0 after lost signal to thread 20385
W20250114 08:57:51.551131 21046 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca860 after lost signal to thread 20957
W20250114 08:57:51.551908 21046 debug-util.cc:398] Leaking SignalData structure 0x7b08001d12c0 after lost signal to thread 21049
W20250114 08:57:51.552732 21046 debug-util.cc:398] Leaking SignalData structure 0x7b08000d59a0 after lost signal to thread 21124
W20250114 08:57:51.553627 21046 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4de0 after lost signal to thread 21198
I20250114 08:57:52.470399 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:52.479115 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:52.479715 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:53.480901 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:53.489849 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:53.490486 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:54.491528 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:54.496857 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:54.497253 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:55.497972 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:55.503132 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:57:55.503531 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:57:56.532859 20382 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20250114 08:57:56.537396 21174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58560
I20250114 08:57:56.538118 21174 raft_consensus.cc:604] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:56.550249 21174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58560
I20250114 08:57:56.550760 21174 raft_consensus.cc:604] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:56.555240 21099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:56016
I20250114 08:57:56.555687 21099 raft_consensus.cc:604] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:56.559796 21099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:56016
I20250114 08:57:56.560236 21099 raft_consensus.cc:604] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:56.564352 21099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
mode: GRACEFUL
new_leader_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:56016
I20250114 08:57:56.564811 21099 raft_consensus.cc:604] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Received request to transfer leadership to TS 26d40bfe96134eb0b1478d2664b2086e
I20250114 08:57:56.783828 21353 raft_consensus.cc:988] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.784253 21353 raft_consensus.cc:1076] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.785703 21024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58142
I20250114 08:57:56.786177 21024 raft_consensus.cc:491] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:56.786540 21024 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.792448 21024 raft_consensus.cc:513] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:56.794238 21024 leader_election.cc:290] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 3 election: Requested vote from peers 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:57:56.794991 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:56.795186 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23fe994c2d4a818f29a1c9c4caf4c3" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:56.795759 21174 raft_consensus.cc:3049] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:56.795789 21099 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.796080 21174 raft_consensus.cc:738] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:56.796748 21174 consensus_queue.cc:260] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:56.797646 21174 raft_consensus.cc:3054] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.800364 21099 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 3.
I20250114 08:57:56.801451 20990 leader_election.cc:304] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:56.802184 21351 raft_consensus.cc:2798] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:56.802659 21351 raft_consensus.cc:695] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 3 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:56.802978 21174 raft_consensus.cc:2463] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 3.
I20250114 08:57:56.803623 21351 consensus_queue.cc:237] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } }
I20250114 08:57:56.813346 20922 catalog_manager.cc:5526] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 2 to 3, leader changed from 8905c307b2d04181a6333a806893cbdc (127.19.231.131) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:56.846091 21346 raft_consensus.cc:988] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.846642 21352 raft_consensus.cc:1076] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.847987 21024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58144
I20250114 08:57:56.848524 21024 raft_consensus.cc:491] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:56.848834 21024 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.853777 21024 raft_consensus.cc:513] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.855793 21024 leader_election.cc:290] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 3 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:56.856693 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:56.856870 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707dcb1fdaf64a2e9f926b3c0c746c55" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:56.857159 21099 raft_consensus.cc:3049] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:56.857334 21174 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.857453 21099 raft_consensus.cc:738] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:56.858148 21099 consensus_queue.cc:260] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.859045 21099 raft_consensus.cc:3054] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.861660 21174 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 3.
I20250114 08:57:56.862555 20990 leader_election.cc:304] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:56.863152 21351 raft_consensus.cc:2798] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:56.863373 21099 raft_consensus.cc:2463] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 3.
I20250114 08:57:56.863597 21351 raft_consensus.cc:695] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 3 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:56.864552 21351 consensus_queue.cc:237] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.871235 20922 catalog_manager.cc:5526] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 2 to 3, leader changed from 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:56.890388 21349 raft_consensus.cc:988] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.890825 21353 raft_consensus.cc:1076] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.892191 21024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58142
I20250114 08:57:56.892695 21024 raft_consensus.cc:491] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:56.892964 21024 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:56.897922 21024 raft_consensus.cc:513] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.898953 21346 raft_consensus.cc:988] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.899423 21352 raft_consensus.cc:1076] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:56.899791 21024 leader_election.cc:290] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 4 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:56.900332 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:56.900898 21025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58144
I20250114 08:57:56.901352 21099 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:56.901415 21025 raft_consensus.cc:491] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:56.901417 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c51502140b4ab691481d101afb0f6c" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:56.901904 21025 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.902112 21174 raft_consensus.cc:3049] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:57:56.902410 21174 raft_consensus.cc:738] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:57:56.903239 21174 consensus_queue.cc:260] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.904412 21174 raft_consensus.cc:3054] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:57:56.906374 21099 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 4.
I20250114 08:57:56.907516 20990 leader_election.cc:304] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 26d40bfe96134eb0b1478d2664b2086e; no voters:
I20250114 08:57:56.908197 21351 raft_consensus.cc:2798] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:57:56.908627 21351 raft_consensus.cc:695] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 4 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:56.909507 21351 consensus_queue.cc:237] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.908556 21025 raft_consensus.cc:513] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.910171 21174 raft_consensus.cc:2463] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 4.
I20250114 08:57:56.911538 21025 leader_election.cc:290] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 3 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:56.913002 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:56.913424 21174 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.913314 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69b04a506c84abda09636ce474bb7ce" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:56.913988 21099 raft_consensus.cc:3049] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:57:56.914392 21099 raft_consensus.cc:738] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:56.915158 21099 consensus_queue.cc:260] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.916101 21099 raft_consensus.cc:3054] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:57:56.917954 21174 raft_consensus.cc:2463] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 3.
I20250114 08:57:56.918792 20990 leader_election.cc:304] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:56.919603 21344 raft_consensus.cc:2798] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:57:56.920925 21344 raft_consensus.cc:695] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 3 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:56.921697 21099 raft_consensus.cc:2463] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 3.
I20250114 08:57:56.921851 21344 consensus_queue.cc:237] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:56.923588 20922 catalog_manager.cc:5526] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 3 to 4, leader changed from 8905c307b2d04181a6333a806893cbdc (127.19.231.131) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:56.937031 20922 catalog_manager.cc:5526] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 2 to 3, leader changed from 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:57.075922 21346 raft_consensus.cc:988] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: : Instructing follower 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:57.076455 21352 raft_consensus.cc:1076] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Signalling peer 26d40bfe96134eb0b1478d2664b2086e to start an election
I20250114 08:57:57.077946 21025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
from {username='slave'} at 127.0.0.1:58144
I20250114 08:57:57.078405 21025 raft_consensus.cc:491] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:57:57.078670 21025 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:57.082854 21025 raft_consensus.cc:513] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:57.084204 21025 leader_election.cc:290] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Requested vote from peers 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:57:57.085078 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:57:57.085243 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "26d40bfe96134eb0b1478d2664b2086e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:57:57.085546 21099 raft_consensus.cc:3049] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:57:57.085734 21174 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:57.085845 21099 raft_consensus.cc:738] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:57:57.086577 21099 consensus_queue.cc:260] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:57.087499 21099 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:57:57.090283 21174 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:57.091233 20990 leader_election.cc:304] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d40bfe96134eb0b1478d2664b2086e, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:57:57.091958 21099 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d40bfe96134eb0b1478d2664b2086e in term 2.
I20250114 08:57:57.092044 21344 raft_consensus.cc:2798] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:57:57.092592 21344 raft_consensus.cc:695] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Leader. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:57:57.093442 21344 consensus_queue.cc:237] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:57:57.100581 20922 catalog_manager.cc:5526] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e reported cstate change: term changed from 1 to 2, leader changed from 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130) to 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "26d40bfe96134eb0b1478d2664b2086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:57:57.289064 21099 raft_consensus.cc:1270] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 4 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:57.290210 21344 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:57.300791 21174 raft_consensus.cc:1270] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 4 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:57:57.301980 21344 consensus_queue.cc:1035] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:57:57.314299 21174 raft_consensus.cc:1270] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:57.315419 21344 consensus_queue.cc:1035] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:57.323617 21099 raft_consensus.cc:1270] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:57.325196 21351 consensus_queue.cc:1035] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:57.392243 21099 raft_consensus.cc:1270] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:57.393450 21351 consensus_queue.cc:1035] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:57.401747 21174 raft_consensus.cc:1270] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:57.403980 21372 consensus_queue.cc:1035] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:57:57.483556 21099 raft_consensus.cc:1270] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:57.484858 21361 consensus_queue.cc:1035] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:57.485983 21174 raft_consensus.cc:1270] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:57:57.487109 21361 consensus_queue.cc:1035] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:57:57.583220 21174 raft_consensus.cc:1270] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:57.584380 21372 consensus_queue.cc:1035] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:57.588533 21099 raft_consensus.cc:1270] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Refusing update from remote peer 26d40bfe96134eb0b1478d2664b2086e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:57:57.589905 21372 consensus_queue.cc:1035] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:57:59.902623 20382 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20250114 08:57:59.903599 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:57:59.911731 21025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "060a086354884b0786d31a6d67b07f81"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58166
I20250114 08:57:59.912204 21025 raft_consensus.cc:604] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Received request to transfer leadership to TS 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:59.914297 21025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58166
I20250114 08:57:59.914724 21025 raft_consensus.cc:604] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Received request to transfer leadership to TS 8905c307b2d04181a6333a806893cbdc
I20250114 08:57:59.916500 21025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:58166
I20250114 08:57:59.916894 21025 raft_consensus.cc:604] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Received request to transfer leadership to TS 08a5ac200b51404694fa174ebc6fe0d5
I20250114 08:57:59.918589 21025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae"
dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
mode: GRACEFUL
new_leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:58166
I20250114 08:57:59.918988 21025 raft_consensus.cc:604] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Received request to transfer leadership to TS 08a5ac200b51404694fa174ebc6fe0d5
I20250114 08:57:59.919870 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250114 08:57:59.920317 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:00.208600 21382 raft_consensus.cc:988] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:58:00.209303 21351 raft_consensus.cc:1076] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Signalling peer 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:58:00.210891 21173 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "060a086354884b0786d31a6d67b07f81"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58534
I20250114 08:58:00.211544 21173 raft_consensus.cc:491] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:58:00.211920 21173 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.218506 21173 raft_consensus.cc:513] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.221208 21173 leader_election.cc:290] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 3 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:58:00.222219 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:58:00.222527 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060a086354884b0786d31a6d67b07f81" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:58:00.223109 21025 raft_consensus.cc:3049] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:00.223176 21100 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.223526 21025 raft_consensus.cc:738] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:58:00.224381 21025 consensus_queue.cc:260] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.225991 21025 raft_consensus.cc:3054] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.231185 21100 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 3.
I20250114 08:58:00.232506 21139 leader_election.cc:304] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:58:00.233346 21025 raft_consensus.cc:2463] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 3.
I20250114 08:58:00.233455 21381 raft_consensus.cc:2798] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:00.234691 21381 raft_consensus.cc:695] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:58:00.235718 21381 consensus_queue.cc:237] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.246183 21382 raft_consensus.cc:988] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:58:00.246675 21351 raft_consensus.cc:1076] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Signalling peer 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:58:00.248296 21100 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:56006
I20250114 08:58:00.248814 21100 raft_consensus.cc:491] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:58:00.249123 21100 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.249631 20922 catalog_manager.cc:5526] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 2 to 3, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 3 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:00.254287 21100 raft_consensus.cc:513] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.256843 21100 leader_election.cc:290] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 3 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:58:00.257654 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:58:00.258060 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1e12f111846c7968bcd95be8ebc5f" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:58:00.258303 21025 raft_consensus.cc:3049] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:00.258654 21025 raft_consensus.cc:738] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:58:00.258699 21173 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.259641 21025 consensus_queue.cc:260] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.261161 21025 raft_consensus.cc:3054] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.266959 21173 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 3.
I20250114 08:58:00.268268 21065 leader_election.cc:304] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:58:00.269073 21352 raft_consensus.cc:2798] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:00.269483 21025 raft_consensus.cc:2463] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 3.
I20250114 08:58:00.270504 21352 raft_consensus.cc:695] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:58:00.271435 21352 consensus_queue.cc:237] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.277590 21382 raft_consensus.cc:988] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:58:00.278218 21351 raft_consensus.cc:1076] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Signalling peer 08a5ac200b51404694fa174ebc6fe0d5 to start an election
I20250114 08:58:00.279902 21100 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae"
dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
from {username='slave'} at 127.0.0.1:56006
I20250114 08:58:00.280625 21100 raft_consensus.cc:491] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:58:00.281006 21100 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.279940 20922 catalog_manager.cc:5526] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 2 to 3, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 3 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:00.286550 21100 raft_consensus.cc:513] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.288280 21100 leader_election.cc:290] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 3 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 8905c307b2d04181a6333a806893cbdc (127.19.231.131:40787)
I20250114 08:58:00.288928 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:58:00.289479 21025 raft_consensus.cc:3049] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:00.289697 21025 raft_consensus.cc:738] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:58:00.289719 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d5ba27eb7e4d2e9aea1fff2b0ab1ae" candidate_uuid: "08a5ac200b51404694fa174ebc6fe0d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8905c307b2d04181a6333a806893cbdc"
I20250114 08:58:00.290398 21173 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.290426 21025 consensus_queue.cc:260] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.291716 21025 raft_consensus.cc:3054] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.295423 21173 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 3.
I20250114 08:58:00.296360 21025 raft_consensus.cc:2463] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a5ac200b51404694fa174ebc6fe0d5 in term 3.
I20250114 08:58:00.296875 21065 leader_election.cc:304] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:58:00.297504 21352 raft_consensus.cc:2798] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:00.297864 21352 raft_consensus.cc:695] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 LEADER]: Becoming Leader. State: Replica: 08a5ac200b51404694fa174ebc6fe0d5, State: Running, Role: LEADER
I20250114 08:58:00.298568 21352 consensus_queue.cc:237] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.305370 20922 catalog_manager.cc:5526] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 reported cstate change: term changed from 2 to 3, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130). New cstate: current_term: 3 leader_uuid: "08a5ac200b51404694fa174ebc6fe0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:00.382042 21382 raft_consensus.cc:988] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: : Instructing follower 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:58:00.382537 21351 raft_consensus.cc:1076] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Signalling peer 8905c307b2d04181a6333a806893cbdc to start an election
I20250114 08:58:00.383913 21173 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11"
dest_uuid: "8905c307b2d04181a6333a806893cbdc"
from {username='slave'} at 127.0.0.1:58534
I20250114 08:58:00.384424 21173 raft_consensus.cc:491] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:58:00.384722 21173 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.390777 21173 raft_consensus.cc:513] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.392838 21173 leader_election.cc:290] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 3 election: Requested vote from peers 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059), 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771)
I20250114 08:58:00.393919 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26d40bfe96134eb0b1478d2664b2086e"
I20250114 08:58:00.394205 21100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13273a9f79d14d13b2569c8e082f7a11" candidate_uuid: "8905c307b2d04181a6333a806893cbdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08a5ac200b51404694fa174ebc6fe0d5"
I20250114 08:58:00.394469 21025 raft_consensus.cc:3049] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:00.394740 21025 raft_consensus.cc:738] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26d40bfe96134eb0b1478d2664b2086e, State: Running, Role: LEADER
I20250114 08:58:00.394764 21100 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.395434 21025 consensus_queue.cc:260] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.396332 21025 raft_consensus.cc:3054] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:00.399518 21100 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 3.
I20250114 08:58:00.400431 21139 leader_election.cc:304] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a5ac200b51404694fa174ebc6fe0d5, 8905c307b2d04181a6333a806893cbdc; no voters:
I20250114 08:58:00.400616 21025 raft_consensus.cc:2463] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8905c307b2d04181a6333a806893cbdc in term 3.
I20250114 08:58:00.401146 21381 raft_consensus.cc:2798] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:00.401547 21381 raft_consensus.cc:695] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Becoming Leader. State: Replica: 8905c307b2d04181a6333a806893cbdc, State: Running, Role: LEADER
I20250114 08:58:00.402357 21381 consensus_queue.cc:237] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } }
I20250114 08:58:00.408967 20922 catalog_manager.cc:5526] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc reported cstate change: term changed from 2 to 3, leader changed from 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129) to 8905c307b2d04181a6333a806893cbdc (127.19.231.131). New cstate: current_term: 3 leader_uuid: "8905c307b2d04181a6333a806893cbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:00.678223 21173 raft_consensus.cc:1270] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.679920 21392 consensus_queue.cc:1035] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250114 08:58:00.684038 21121 debug-util.cc:398] Leaking SignalData structure 0x7b080011b060 after lost signal to thread 20385
W20250114 08:58:00.685163 21121 debug-util.cc:398] Leaking SignalData structure 0x7b080011b740 after lost signal to thread 20957
W20250114 08:58:00.686419 21121 debug-util.cc:398] Leaking SignalData structure 0x7b08001bc6a0 after lost signal to thread 21049
W20250114 08:58:00.687374 21121 debug-util.cc:398] Leaking SignalData structure 0x7b0800058ec0 after lost signal to thread 21124
W20250114 08:58:00.688586 21121 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3c40 after lost signal to thread 21198
I20250114 08:58:00.700368 21025 raft_consensus.cc:1270] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.701934 21352 consensus_queue.cc:1035] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:00.804065 21100 raft_consensus.cc:1270] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.805382 21381 consensus_queue.cc:1035] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:00.807898 21100 raft_consensus.cc:1270] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.809401 21381 consensus_queue.cc:1035] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a5ac200b51404694fa174ebc6fe0d5" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:00.818018 21024 raft_consensus.cc:1270] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.821467 21386 consensus_queue.cc:1035] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:00.826146 21024 raft_consensus.cc:1270] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Refusing update from remote peer 8905c307b2d04181a6333a806893cbdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.827993 21406 consensus_queue.cc:1035] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:00.867607 21024 raft_consensus.cc:1270] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.868985 21392 consensus_queue.cc:1035] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d40bfe96134eb0b1478d2664b2086e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:00.880071 21173 raft_consensus.cc:1270] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Refusing update from remote peer 08a5ac200b51404694fa174ebc6fe0d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:00.881852 21392 consensus_queue.cc:1035] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8905c307b2d04181a6333a806893cbdc" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:00.921267 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:00.926690 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:00.927125 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:01.927915 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:01.933730 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:01.934139 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:02.935137 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:02.944262 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:02.944892 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:03.945776 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:03.951022 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:03.951458 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:04.952344 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:04.957901 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:04.958343 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:05.959170 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:05.964401 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:05.964902 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:06.965735 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:06.970925 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:06.971385 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:07.972215 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:07.977466 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:07.977876 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:08.978713 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:08.986178 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 08:58:08.986586 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:10.015059 20382 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20250114 08:58:10.015587 20382 tablet_server.cc:178] TabletServer@127.19.231.129:0 shutting down...
W20250114 08:58:10.062952 21139 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20250114 08:58:10.065414 20382 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 08:58:10.066766 20382 tablet_replica.cc:331] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
W20250114 08:58:10.067783 21139 consensus_peers.cc:487] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc -> Peer 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059): Couldn't send request to peer 26d40bfe96134eb0b1478d2664b2086e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:10.068259 20382 raft_consensus.cc:2238] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.069904 20382 raft_consensus.cc:2267] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.072185 20382 tablet_replica.cc:331] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.072947 20382 raft_consensus.cc:2238] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.073561 20382 raft_consensus.cc:2267] T f559c3fc5ad7468f9b13ba6572cc25cd P 26d40bfe96134eb0b1478d2664b2086e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.075619 20382 tablet_replica.cc:331] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.076311 20382 raft_consensus.cc:2238] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.076938 20382 raft_consensus.cc:2267] T 13273a9f79d14d13b2569c8e082f7a11 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.078882 20382 tablet_replica.cc:331] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.079403 20382 raft_consensus.cc:2238] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.080910 20382 raft_consensus.cc:2267] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.083177 20382 tablet_replica.cc:331] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.083774 20382 raft_consensus.cc:2238] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.085065 20382 raft_consensus.cc:2267] T a69b04a506c84abda09636ce474bb7ce P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.087852 20382 tablet_replica.cc:331] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.088383 20382 raft_consensus.cc:2238] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.088944 20382 raft_consensus.cc:2267] T 11dd206a2b6246a1935e5b943ab23baf P 26d40bfe96134eb0b1478d2664b2086e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.090962 20382 tablet_replica.cc:331] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.091490 20382 raft_consensus.cc:2238] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.092073 20382 raft_consensus.cc:2267] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.094066 20382 tablet_replica.cc:331] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.094647 20382 raft_consensus.cc:2238] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.095182 20382 raft_consensus.cc:2267] T 060a086354884b0786d31a6d67b07f81 P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.097118 20382 tablet_replica.cc:331] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.097705 20382 raft_consensus.cc:2238] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 4 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.099606 20382 raft_consensus.cc:2267] T a9c51502140b4ab691481d101afb0f6c P 26d40bfe96134eb0b1478d2664b2086e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.102701 20382 tablet_replica.cc:331] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e: stopping tablet replica
I20250114 08:58:10.103171 20382 raft_consensus.cc:2238] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.103636 20382 raft_consensus.cc:2267] T 58d1e12f111846c7968bcd95be8ebc5f P 26d40bfe96134eb0b1478d2664b2086e [term 3 FOLLOWER]: Raft consensus is shut down!
W20250114 08:58:10.142164 21065 consensus_peers.cc:487] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 -> Peer 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059): Couldn't send request to peer 26d40bfe96134eb0b1478d2664b2086e. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.129:36059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:10.155434 20382 tablet_server.cc:195] TabletServer@127.19.231.129:0 shutdown complete.
I20250114 08:58:10.176913 20382 tablet_server.cc:178] TabletServer@127.19.231.130:0 shutting down...
I20250114 08:58:10.210626 20382 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20250114 08:58:10.212579 21139 consensus_peers.cc:487] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc -> Peer 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059): Couldn't send request to peer 26d40bfe96134eb0b1478d2664b2086e. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.129:36059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:10.213707 20382 tablet_replica.cc:331] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.215230 20382 raft_consensus.cc:2238] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.215946 20382 raft_consensus.cc:2267] T 11dd206a2b6246a1935e5b943ab23baf P 08a5ac200b51404694fa174ebc6fe0d5 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250114 08:58:10.216799 21139 consensus_peers.cc:487] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc -> Peer 08a5ac200b51404694fa174ebc6fe0d5 (127.19.231.130:45771): Couldn't send request to peer 08a5ac200b51404694fa174ebc6fe0d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250114 08:58:10.216816 21065 consensus_peers.cc:487] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 -> Peer 26d40bfe96134eb0b1478d2664b2086e (127.19.231.129:36059): Couldn't send request to peer 26d40bfe96134eb0b1478d2664b2086e. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.129:36059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:10.218742 20382 tablet_replica.cc:331] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.219578 20382 raft_consensus.cc:2238] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.220072 20382 raft_consensus.cc:2267] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.222009 20382 tablet_replica.cc:331] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.222581 20382 raft_consensus.cc:2238] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.223639 20382 raft_consensus.cc:2267] T f559c3fc5ad7468f9b13ba6572cc25cd P 08a5ac200b51404694fa174ebc6fe0d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.225582 20382 tablet_replica.cc:331] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.226099 20382 raft_consensus.cc:2238] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.226577 20382 raft_consensus.cc:2267] T 13273a9f79d14d13b2569c8e082f7a11 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.229104 20382 tablet_replica.cc:331] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.229768 20382 raft_consensus.cc:2238] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.230434 20382 raft_consensus.cc:2267] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.232580 20382 tablet_replica.cc:331] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.233121 20382 raft_consensus.cc:2238] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.233664 20382 raft_consensus.cc:2267] T a9c51502140b4ab691481d101afb0f6c P 08a5ac200b51404694fa174ebc6fe0d5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.235507 20382 tablet_replica.cc:331] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.236044 20382 raft_consensus.cc:2238] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.236995 20382 raft_consensus.cc:2267] T 58d1e12f111846c7968bcd95be8ebc5f P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.238883 20382 tablet_replica.cc:331] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.239390 20382 raft_consensus.cc:2238] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.239867 20382 raft_consensus.cc:2267] T a69b04a506c84abda09636ce474bb7ce P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.241950 20382 tablet_replica.cc:331] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.242692 20382 raft_consensus.cc:2238] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.243306 20382 raft_consensus.cc:2267] T 060a086354884b0786d31a6d67b07f81 P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.245419 20382 tablet_replica.cc:331] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5: stopping tablet replica
I20250114 08:58:10.245877 20382 raft_consensus.cc:2238] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.246619 20382 raft_consensus.cc:2267] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 08a5ac200b51404694fa174ebc6fe0d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.294276 20382 tablet_server.cc:195] TabletServer@127.19.231.130:0 shutdown complete.
I20250114 08:58:10.315042 20382 tablet_server.cc:178] TabletServer@127.19.231.131:0 shutting down...
I20250114 08:58:10.340903 20382 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 08:58:10.341630 20382 tablet_replica.cc:331] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.342306 20382 raft_consensus.cc:2238] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.342835 20382 raft_consensus.cc:2267] T 707dcb1fdaf64a2e9f926b3c0c746c55 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.344730 20382 tablet_replica.cc:331] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.345216 20382 raft_consensus.cc:2238] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.345670 20382 raft_consensus.cc:2267] T 8e23fe994c2d4a818f29a1c9c4caf4c3 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.347405 20382 tablet_replica.cc:331] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.347896 20382 raft_consensus.cc:2238] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.348790 20382 raft_consensus.cc:2267] T 13273a9f79d14d13b2569c8e082f7a11 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.350654 20382 tablet_replica.cc:331] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.351161 20382 raft_consensus.cc:2238] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.351586 20382 raft_consensus.cc:2267] T f559c3fc5ad7468f9b13ba6572cc25cd P 8905c307b2d04181a6333a806893cbdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.353298 20382 tablet_replica.cc:331] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.353775 20382 raft_consensus.cc:2238] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 2 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.354581 20382 raft_consensus.cc:2267] T 11dd206a2b6246a1935e5b943ab23baf P 8905c307b2d04181a6333a806893cbdc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.356243 20382 tablet_replica.cc:331] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.356758 20382 raft_consensus.cc:2238] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.357214 20382 raft_consensus.cc:2267] T 58d1e12f111846c7968bcd95be8ebc5f P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.358846 20382 tablet_replica.cc:331] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.359308 20382 raft_consensus.cc:2238] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.359745 20382 raft_consensus.cc:2267] T a9c51502140b4ab691481d101afb0f6c P 8905c307b2d04181a6333a806893cbdc [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.361495 20382 tablet_replica.cc:331] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.361977 20382 raft_consensus.cc:2238] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.362461 20382 raft_consensus.cc:2267] T a69b04a506c84abda09636ce474bb7ce P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.364318 20382 tablet_replica.cc:331] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.364816 20382 raft_consensus.cc:2238] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:10.365208 20382 raft_consensus.cc:2267] T c8d5ba27eb7e4d2e9aea1fff2b0ab1ae P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.366757 20382 tablet_replica.cc:331] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc: stopping tablet replica
I20250114 08:58:10.367159 20382 raft_consensus.cc:2238] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 3 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.368074 20382 raft_consensus.cc:2267] T 060a086354884b0786d31a6d67b07f81 P 8905c307b2d04181a6333a806893cbdc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.397377 20382 tablet_server.cc:195] TabletServer@127.19.231.131:0 shutdown complete.
I20250114 08:58:10.417192 20382 master.cc:537] Master@127.19.231.190:45087 shutting down...
I20250114 08:58:10.442648 20382 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 1 LEADER]: Raft consensus shutting down.
I20250114 08:58:10.443248 20382 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:10.443583 20382 tablet_replica.cc:331] T 00000000000000000000000000000000 P 28b0eb2db0c5487c9e24094027020b29: stopping tablet replica
I20250114 08:58:10.462522 20382 master.cc:559] Master@127.19.231.190:45087 shutdown complete.
08a5ac200b51404694fa174ebc6fe0d5 4
26d40bfe96134eb0b1478d2664b2086e 5
8905c307b2d04181a6333a806893cbdc 1
08a5ac200b51404694fa174ebc6fe0d5 4
26d40bfe96134eb0b1478d2664b2086e 3
8905c307b2d04181a6333a806893cbdc 3
08a5ac200b51404694fa174ebc6fe0d5 1
26d40bfe96134eb0b1478d2664b2086e 8
8905c307b2d04181a6333a806893cbdc 1
08a5ac200b51404694fa174ebc6fe0d5 3
26d40bfe96134eb0b1478d2664b2086e 4
8905c307b2d04181a6333a806893cbdc 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29720 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250114 08:58:10.504006 20382 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.231.190:44737
I20250114 08:58:10.505194 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:58:10.511618 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.512552 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.513535 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:58:10.514339 20382 server_base.cc:1034] running on GCE node
I20250114 08:58:10.515288 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:58:10.515508 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:58:10.515654 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845090515636 us; error 0 us; skew 500 ppm
I20250114 08:58:10.516183 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:58:10.523728 20382 webserver.cc:458] Webserver started at http://127.19.231.190:33785/ using document root <none> and password file <none>
I20250114 08:58:10.524245 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:58:10.524441 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:58:10.524758 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:58:10.525967 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/master-0-root/instance:
uuid: "221e58f7b595419f9fb78c5c0e6207b5"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.531163 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250114 08:58:10.534601 21436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.535452 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 08:58:10.535717 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/master-0-root
uuid: "221e58f7b595419f9fb78c5c0e6207b5"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.535964 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:58:10.556618 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:58:10.557852 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:58:10.596099 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.190:44737
I20250114 08:58:10.596194 21487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.190:44737 every 8 connection(s)
I20250114 08:58:10.599968 21488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:10.609649 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Bootstrap starting.
I20250114 08:58:10.614293 21488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:10.618562 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: No bootstrap required, opened a new log
I20250114 08:58:10.620721 21488 raft_consensus.cc:357] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221e58f7b595419f9fb78c5c0e6207b5" member_type: VOTER }
I20250114 08:58:10.621222 21488 raft_consensus.cc:383] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:10.621472 21488 raft_consensus.cc:738] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221e58f7b595419f9fb78c5c0e6207b5, State: Initialized, Role: FOLLOWER
I20250114 08:58:10.622030 21488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221e58f7b595419f9fb78c5c0e6207b5" member_type: VOTER }
I20250114 08:58:10.622481 21488 raft_consensus.cc:397] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 08:58:10.622717 21488 raft_consensus.cc:491] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 08:58:10.622992 21488 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:10.627156 21488 raft_consensus.cc:513] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221e58f7b595419f9fb78c5c0e6207b5" member_type: VOTER }
I20250114 08:58:10.627699 21488 leader_election.cc:304] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221e58f7b595419f9fb78c5c0e6207b5; no voters:
I20250114 08:58:10.628844 21488 leader_election.cc:290] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 08:58:10.629284 21491 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:10.630697 21491 raft_consensus.cc:695] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [term 1 LEADER]: Becoming Leader. State: Replica: 221e58f7b595419f9fb78c5c0e6207b5, State: Running, Role: LEADER
I20250114 08:58:10.631436 21491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221e58f7b595419f9fb78c5c0e6207b5" member_type: VOTER }
I20250114 08:58:10.631812 21488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 08:58:10.637676 21492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221e58f7b595419f9fb78c5c0e6207b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221e58f7b595419f9fb78c5c0e6207b5" member_type: VOTER } }
I20250114 08:58:10.638597 21492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [sys.catalog]: This master's current role is: LEADER
I20250114 08:58:10.638489 21493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221e58f7b595419f9fb78c5c0e6207b5. Latest consensus state: current_term: 1 leader_uuid: "221e58f7b595419f9fb78c5c0e6207b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221e58f7b595419f9fb78c5c0e6207b5" member_type: VOTER } }
I20250114 08:58:10.639122 21493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5 [sys.catalog]: This master's current role is: LEADER
I20250114 08:58:10.640645 21499 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 08:58:10.645493 21499 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 08:58:10.648450 20382 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 08:58:10.654446 21499 catalog_manager.cc:1348] Generated new cluster ID: f9a45130b1184e2cb3a71af71526d4a1
I20250114 08:58:10.654759 21499 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 08:58:10.667552 21499 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 08:58:10.668869 21499 catalog_manager.cc:1505] Loading token signing keys...
I20250114 08:58:10.684111 21499 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Generated new TSK 0
I20250114 08:58:10.684737 21499 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 08:58:10.715449 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:58:10.721724 21509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.723338 21510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.724182 21512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:58:10.725186 20382 server_base.cc:1034] running on GCE node
I20250114 08:58:10.726022 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:58:10.726207 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:58:10.726326 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845090726316 us; error 0 us; skew 500 ppm
I20250114 08:58:10.726773 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:58:10.729243 20382 webserver.cc:458] Webserver started at http://127.19.231.129:35283/ using document root <none> and password file <none>
I20250114 08:58:10.729849 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:58:10.730024 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:58:10.730261 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:58:10.731302 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/instance:
uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.735942 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250114 08:58:10.739429 21517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.740150 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 08:58:10.740442 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root
uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.740763 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:58:10.754848 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:58:10.756103 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:58:10.757618 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:58:10.759879 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:58:10.760072 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.760310 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:58:10.760493 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.801230 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.129:45539
I20250114 08:58:10.801366 21579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.129:45539 every 8 connection(s)
I20250114 08:58:10.805773 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:58:10.814085 21584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.815429 21585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.818162 21587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:58:10.819484 20382 server_base.cc:1034] running on GCE node
I20250114 08:58:10.819773 21580 heartbeater.cc:346] Connected to a master server at 127.19.231.190:44737
I20250114 08:58:10.820519 21580 heartbeater.cc:463] Registering TS with master...
I20250114 08:58:10.821177 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 08:58:10.821241 21580 heartbeater.cc:510] Master 127.19.231.190:44737 requested a full tablet report, sending...
W20250114 08:58:10.821416 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:58:10.821635 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845090821617 us; error 0 us; skew 500 ppm
I20250114 08:58:10.822288 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:58:10.823387 21453 ts_manager.cc:194] Registered new tserver with Master: 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:10.825327 20382 webserver.cc:458] Webserver started at http://127.19.231.130:37411/ using document root <none> and password file <none>
I20250114 08:58:10.825477 21453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55864
I20250114 08:58:10.825855 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:58:10.826043 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:58:10.826256 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:58:10.827467 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root/instance:
uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.831902 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250114 08:58:10.835347 21592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.836117 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 08:58:10.836422 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root
uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.836745 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:58:10.852543 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:58:10.853812 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:58:10.855429 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:58:10.857801 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:58:10.857997 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.858273 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:58:10.858422 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.897749 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.130:36891
I20250114 08:58:10.897843 21654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.130:36891 every 8 connection(s)
I20250114 08:58:10.902458 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:58:10.909929 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.911499 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:10.912583 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:58:10.914008 20382 server_base.cc:1034] running on GCE node
I20250114 08:58:10.914286 21655 heartbeater.cc:346] Connected to a master server at 127.19.231.190:44737
I20250114 08:58:10.914718 21655 heartbeater.cc:463] Registering TS with master...
I20250114 08:58:10.915113 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:58:10.915392 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:58:10.915630 21655 heartbeater.cc:510] Master 127.19.231.190:44737 requested a full tablet report, sending...
I20250114 08:58:10.915642 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845090915626 us; error 0 us; skew 500 ppm
I20250114 08:58:10.916700 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:58:10.917927 21453 ts_manager.cc:194] Registered new tserver with Master: d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:10.919355 20382 webserver.cc:458] Webserver started at http://127.19.231.131:35609/ using document root <none> and password file <none>
I20250114 08:58:10.919497 21453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55874
I20250114 08:58:10.919884 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:58:10.920097 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:58:10.920352 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:58:10.921952 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root/instance:
uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.926419 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250114 08:58:10.929706 21666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.930600 20382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250114 08:58:10.930941 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root
uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
format_stamp: "Formatted at 2025-01-14 08:58:10 on dist-test-slave-4wc2"
I20250114 08:58:10.931217 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:58:10.950960 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:58:10.952214 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:58:10.953711 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:58:10.955986 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:58:10.956207 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:10.956444 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:58:10.956622 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250114 08:58:10.995550 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.131:37485
I20250114 08:58:10.995656 21728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.131:37485 every 8 connection(s)
I20250114 08:58:11.012382 21729 heartbeater.cc:346] Connected to a master server at 127.19.231.190:44737
I20250114 08:58:11.012780 21729 heartbeater.cc:463] Registering TS with master...
I20250114 08:58:11.013484 21729 heartbeater.cc:510] Master 127.19.231.190:44737 requested a full tablet report, sending...
I20250114 08:58:11.015285 21453 ts_manager.cc:194] Registered new tserver with Master: cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:11.015699 20382 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017067026s
I20250114 08:58:11.017017 21453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55888
I20250114 08:58:11.018772 20382 test_util.cc:274] Using random seed: -1028777984
I20250114 08:58:11.042963 21453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55898:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 08:58:11.049103 21453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 08:58:11.257774 21694 tablet_service.cc:1467] Processing CreateTablet for tablet 7050054fd28749bca69cd4c2e1336d7b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250114 08:58:11.259279 21690 tablet_service.cc:1467] Processing CreateTablet for tablet 073e9192e9cb48728d4dd8027da50f60 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250114 08:58:11.260522 21693 tablet_service.cc:1467] Processing CreateTablet for tablet 426decde999b4722a874569acae5664f (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250114 08:58:11.261561 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6414eaae2e412fa959acd24b4f7368 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250114 08:58:11.262032 21613 tablet_service.cc:1467] Processing CreateTablet for tablet fa184c9a07ba40518b2e4d61a40e4da2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250114 08:58:11.262974 21692 tablet_service.cc:1467] Processing CreateTablet for tablet ec8afddecbe8465193b684ed870589f0 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250114 08:58:11.263635 21613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa184c9a07ba40518b2e4d61a40e4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.263247 21689 tablet_service.cc:1467] Processing CreateTablet for tablet a03f2f2f70724b85b8b23ed7bc72fbe1 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250114 08:58:11.263923 21687 tablet_service.cc:1467] Processing CreateTablet for tablet fa184c9a07ba40518b2e4d61a40e4da2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250114 08:58:11.259299 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7050054fd28749bca69cd4c2e1336d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.265295 21691 tablet_service.cc:1467] Processing CreateTablet for tablet b7929f46385a41b0acb9e10a87605e22 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250114 08:58:11.265719 21688 tablet_service.cc:1467] Processing CreateTablet for tablet a5d394e52b4f4ee2be4e4f4a32ce8547 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250114 08:58:11.266795 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7929f46385a41b0acb9e10a87605e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.267458 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a03f2f2f70724b85b8b23ed7bc72fbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.275552 21617 tablet_service.cc:1467] Processing CreateTablet for tablet b7929f46385a41b0acb9e10a87605e22 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250114 08:58:11.275952 21541 tablet_service.cc:1467] Processing CreateTablet for tablet 073e9192e9cb48728d4dd8027da50f60 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250114 08:58:11.284677 21538 tablet_service.cc:1467] Processing CreateTablet for tablet fa184c9a07ba40518b2e4d61a40e4da2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250114 08:58:11.294683 21541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073e9192e9cb48728d4dd8027da50f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.280336 21539 tablet_service.cc:1467] Processing CreateTablet for tablet a5d394e52b4f4ee2be4e4f4a32ce8547 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250114 08:58:11.292605 21536 tablet_service.cc:1467] Processing CreateTablet for tablet 125d011b631240b8bd1138f8cd52ec2e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250114 08:58:11.274746 21542 tablet_service.cc:1467] Processing CreateTablet for tablet b7929f46385a41b0acb9e10a87605e22 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250114 08:58:11.297727 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125d011b631240b8bd1138f8cd52ec2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.269114 21620 tablet_service.cc:1467] Processing CreateTablet for tablet 7050054fd28749bca69cd4c2e1336d7b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250114 08:58:11.273521 21543 tablet_service.cc:1467] Processing CreateTablet for tablet ec8afddecbe8465193b684ed870589f0 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250114 08:58:11.274144 21545 tablet_service.cc:1467] Processing CreateTablet for tablet 7050054fd28749bca69cd4c2e1336d7b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250114 08:58:11.300190 21543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8afddecbe8465193b684ed870589f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.275401 21540 tablet_service.cc:1467] Processing CreateTablet for tablet a03f2f2f70724b85b8b23ed7bc72fbe1 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250114 08:58:11.302363 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a03f2f2f70724b85b8b23ed7bc72fbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.280884 21612 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6414eaae2e412fa959acd24b4f7368 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250114 08:58:11.289891 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6414eaae2e412fa959acd24b4f7368 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250114 08:58:11.272300 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 426decde999b4722a874569acae5664f (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250114 08:58:11.281476 21618 tablet_service.cc:1467] Processing CreateTablet for tablet ec8afddecbe8465193b684ed870589f0 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250114 08:58:11.305943 21542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7929f46385a41b0acb9e10a87605e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.280052 21614 tablet_service.cc:1467] Processing CreateTablet for tablet a5d394e52b4f4ee2be4e4f4a32ce8547 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250114 08:58:11.268361 21690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073e9192e9cb48728d4dd8027da50f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.307085 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6414eaae2e412fa959acd24b4f7368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.277967 21619 tablet_service.cc:1467] Processing CreateTablet for tablet 426decde999b4722a874569acae5664f (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250114 08:58:11.269781 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 125d011b631240b8bd1138f8cd52ec2e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250114 08:58:11.309303 21692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8afddecbe8465193b684ed870589f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.312837 21686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6414eaae2e412fa959acd24b4f7368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.271742 21615 tablet_service.cc:1467] Processing CreateTablet for tablet a03f2f2f70724b85b8b23ed7bc72fbe1 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250114 08:58:11.274277 21616 tablet_service.cc:1467] Processing CreateTablet for tablet 073e9192e9cb48728d4dd8027da50f60 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250114 08:58:11.318306 21617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7929f46385a41b0acb9e10a87605e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.273015 21685 tablet_service.cc:1467] Processing CreateTablet for tablet 125d011b631240b8bd1138f8cd52ec2e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250114 08:58:11.322052 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125d011b631240b8bd1138f8cd52ec2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.326345 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d394e52b4f4ee2be4e4f4a32ce8547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.328891 21693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426decde999b4722a874569acae5664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.330298 21687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa184c9a07ba40518b2e4d61a40e4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.316982 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa184c9a07ba40518b2e4d61a40e4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.335150 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7050054fd28749bca69cd4c2e1336d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.336817 21539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d394e52b4f4ee2be4e4f4a32ce8547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.361959 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426decde999b4722a874569acae5664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.364707 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d394e52b4f4ee2be4e4f4a32ce8547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.369118 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073e9192e9cb48728d4dd8027da50f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.371114 21619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426decde999b4722a874569acae5664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.373138 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125d011b631240b8bd1138f8cd52ec2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.381196 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a03f2f2f70724b85b8b23ed7bc72fbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.387451 21620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7050054fd28749bca69cd4c2e1336d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.389164 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8afddecbe8465193b684ed870589f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.389920 21612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6414eaae2e412fa959acd24b4f7368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.429546 21689 tablet_service.cc:1467] Processing CreateTablet for tablet 0564bed845d3430e8831ee6428122144 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250114 08:58:11.430964 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0564bed845d3430e8831ee6428122144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.488500 21694 tablet_service.cc:1467] Processing CreateTablet for tablet 80026dc5df8c4456a48f734e1a64b930 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250114 08:58:11.489893 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80026dc5df8c4456a48f734e1a64b930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.491173 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 0564bed845d3430e8831ee6428122144 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250114 08:58:11.497653 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 80026dc5df8c4456a48f734e1a64b930 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250114 08:58:11.500998 21615 tablet_service.cc:1467] Processing CreateTablet for tablet 0564bed845d3430e8831ee6428122144 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250114 08:58:11.502377 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0564bed845d3430e8831ee6428122144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.502892 21619 tablet_service.cc:1467] Processing CreateTablet for tablet 80026dc5df8c4456a48f734e1a64b930 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250114 08:58:11.504237 21619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80026dc5df8c4456a48f734e1a64b930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.510936 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0564bed845d3430e8831ee6428122144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.514503 21692 tablet_service.cc:1467] Processing CreateTablet for tablet 6a451ff14593400ca08caa1b5ed169f2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250114 08:58:11.518205 21693 tablet_service.cc:1467] Processing CreateTablet for tablet addec468f1bd4f19a98a8d5ce1222fc1 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250114 08:58:11.520752 21692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a451ff14593400ca08caa1b5ed169f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.525544 21545 tablet_service.cc:1467] Processing CreateTablet for tablet addec468f1bd4f19a98a8d5ce1222fc1 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250114 08:58:11.527011 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addec468f1bd4f19a98a8d5ce1222fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.529093 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80026dc5df8c4456a48f734e1a64b930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.520367 21688 tablet_service.cc:1467] Processing CreateTablet for tablet 5b52f2d242e445319680653bc6c30fff (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250114 08:58:11.532625 21693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addec468f1bd4f19a98a8d5ce1222fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.533264 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b52f2d242e445319680653bc6c30fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.539908 21539 tablet_service.cc:1467] Processing CreateTablet for tablet 6a451ff14593400ca08caa1b5ed169f2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250114 08:58:11.541661 21616 tablet_service.cc:1467] Processing CreateTablet for tablet 5b52f2d242e445319680653bc6c30fff (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250114 08:58:11.551221 21614 tablet_service.cc:1467] Processing CreateTablet for tablet addec468f1bd4f19a98a8d5ce1222fc1 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250114 08:58:11.553376 21745 tablet_bootstrap.cc:492] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:11.551358 21613 tablet_service.cc:1467] Processing CreateTablet for tablet 6a451ff14593400ca08caa1b5ed169f2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250114 08:58:11.551229 21538 tablet_service.cc:1467] Processing CreateTablet for tablet d19fa6dec61d44618fee387c9b1f2994 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250114 08:58:11.558610 21618 tablet_service.cc:1467] Processing CreateTablet for tablet d19fa6dec61d44618fee387c9b1f2994 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250114 08:58:11.559965 21744 tablet_bootstrap.cc:492] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:11.564635 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b52f2d242e445319680653bc6c30fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.565152 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 5b52f2d242e445319680653bc6c30fff (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250114 08:58:11.567836 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19fa6dec61d44618fee387c9b1f2994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.568487 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addec468f1bd4f19a98a8d5ce1222fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.571883 21746 tablet_bootstrap.cc:492] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:11.572010 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 61f37f8d5b9d4c71b6e852f77fa438e8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250114 08:58:11.572795 21539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a451ff14593400ca08caa1b5ed169f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.573457 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61f37f8d5b9d4c71b6e852f77fa438e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.573469 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19fa6dec61d44618fee387c9b1f2994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.583336 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b52f2d242e445319680653bc6c30fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.584779 21613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a451ff14593400ca08caa1b5ed169f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.594352 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 61f37f8d5b9d4c71b6e852f77fa438e8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250114 08:58:11.595754 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61f37f8d5b9d4c71b6e852f77fa438e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.563124 21686 tablet_service.cc:1467] Processing CreateTablet for tablet d19fa6dec61d44618fee387c9b1f2994 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250114 08:58:11.597643 21686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19fa6dec61d44618fee387c9b1f2994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.599045 21744 tablet_bootstrap.cc:654] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:11.627718 21746 tablet_bootstrap.cc:654] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:11.638648 21745 tablet_bootstrap.cc:654] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:11.643898 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:11.642875 21694 tablet_service.cc:1467] Processing CreateTablet for tablet 61f37f8d5b9d4c71b6e852f77fa438e8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250114 08:58:11.645335 21615 tablet_service.cc:1467] Processing CreateTablet for tablet b3650a91ff4a4213b9091f197bf75fb9 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250114 08:58:11.647230 21688 tablet_service.cc:1467] Processing CreateTablet for tablet b3650a91ff4a4213b9091f197bf75fb9 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250114 08:58:11.649853 21537 tablet_service.cc:1467] Processing CreateTablet for tablet b3650a91ff4a4213b9091f197bf75fb9 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250114 08:58:11.650513 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3650a91ff4a4213b9091f197bf75fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.651151 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3650a91ff4a4213b9091f197bf75fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.645349 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61f37f8d5b9d4c71b6e852f77fa438e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.657110 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3650a91ff4a4213b9091f197bf75fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.668308 21619 tablet_service.cc:1467] Processing CreateTablet for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250114 08:58:11.668783 21693 tablet_service.cc:1467] Processing CreateTablet for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250114 08:58:11.669771 21619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.669988 21693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.670480 21539 tablet_service.cc:1467] Processing CreateTablet for tablet a93e2235a71d41a4a46a1e50fd8c02b7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250114 08:58:11.671746 21539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a93e2235a71d41a4a46a1e50fd8c02b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.674527 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250114 08:58:11.676226 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.704165 21685 tablet_service.cc:1467] Processing CreateTablet for tablet a93e2235a71d41a4a46a1e50fd8c02b7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250114 08:58:11.705763 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a93e2235a71d41a4a46a1e50fd8c02b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.709322 21538 tablet_service.cc:1467] Processing CreateTablet for tablet a1aeae7c29264f839189001364c51746 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250114 08:58:11.711066 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aeae7c29264f839189001364c51746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.711898 21745 tablet_bootstrap.cc:492] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:11.716847 21745 ts_tablet_manager.cc:1397] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.164s user 0.002s sys 0.018s
I20250114 08:58:11.719426 21692 tablet_service.cc:1467] Processing CreateTablet for tablet a1aeae7c29264f839189001364c51746 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250114 08:58:11.719556 21618 tablet_service.cc:1467] Processing CreateTablet for tablet a1aeae7c29264f839189001364c51746 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250114 08:58:11.720824 21692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aeae7c29264f839189001364c51746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.720968 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aeae7c29264f839189001364c51746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.719836 21745 raft_consensus.cc:357] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:11.724623 21745 raft_consensus.cc:383] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:11.725072 21745 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:11.726070 21745 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:11.732511 21611 tablet_service.cc:1467] Processing CreateTablet for tablet a93e2235a71d41a4a46a1e50fd8c02b7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250114 08:58:11.743844 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a93e2235a71d41a4a46a1e50fd8c02b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.766193 21744 tablet_bootstrap.cc:492] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:11.792624 21744 ts_tablet_manager.cc:1397] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.243s user 0.013s sys 0.014s
I20250114 08:58:11.805418 21580 heartbeater.cc:502] Master 127.19.231.190:44737 was elected leader, sending a full tablet report...
I20250114 08:58:11.805124 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 001d4f9f4b5d4124b95dcddb6adac2d6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250114 08:58:11.806479 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 001d4f9f4b5d4124b95dcddb6adac2d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:11.808631 21581 tablet.cc:2367] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 08:58:11.809686 21746 tablet_bootstrap.cc:492] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:11.810194 21746 ts_tablet_manager.cc:1397] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.239s user 0.035s sys 0.036s
I20250114 08:58:11.812336 21613 tablet_service.cc:1467] Processing CreateTablet for tablet 001d4f9f4b5d4124b95dcddb6adac2d6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250114 08:58:11.813629 21613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 001d4f9f4b5d4124b95dcddb6adac2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.796377 21744 raft_consensus.cc:357] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:11.816759 21744 raft_consensus.cc:383] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:11.820596 21744 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:11.804041 21745 ts_tablet_manager.cc:1428] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.087s user 0.015s sys 0.049s
I20250114 08:58:11.829355 21745 tablet_bootstrap.cc:492] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:11.832252 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 82452b55145444529bf0609cd7767adf (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250114 08:58:11.822728 21746 raft_consensus.cc:357] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:11.833154 21746 raft_consensus.cc:383] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:11.833469 21746 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:11.831344 21615 tablet_service.cc:1467] Processing CreateTablet for tablet 82452b55145444529bf0609cd7767adf (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250114 08:58:11.834774 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82452b55145444529bf0609cd7767adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.835371 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4913bb74ad4fab98dd737eb0bd5965 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250114 08:58:11.836612 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4913bb74ad4fab98dd737eb0bd5965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.837883 21746 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:11.824749 21744 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:11.840144 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82452b55145444529bf0609cd7767adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.850133 21693 tablet_service.cc:1467] Processing CreateTablet for tablet 82452b55145444529bf0609cd7767adf (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250114 08:58:11.851542 21693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82452b55145444529bf0609cd7767adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.856922 21655 heartbeater.cc:502] Master 127.19.231.190:44737 was elected leader, sending a full tablet report...
I20250114 08:58:11.858486 21685 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4913bb74ad4fab98dd737eb0bd5965 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250114 08:58:11.860214 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4913bb74ad4fab98dd737eb0bd5965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.860514 21745 tablet_bootstrap.cc:654] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:11.858506 21744 ts_tablet_manager.cc:1428] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.065s user 0.008s sys 0.002s
I20250114 08:58:11.861352 21729 heartbeater.cc:502] Master 127.19.231.190:44737 was elected leader, sending a full tablet report...
I20250114 08:58:11.861480 21545 tablet_service.cc:1467] Processing CreateTablet for tablet abedb4afaad04af9bbbb7d4266fd2632 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250114 08:58:11.862803 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abedb4afaad04af9bbbb7d4266fd2632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.863613 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 001d4f9f4b5d4124b95dcddb6adac2d6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250114 08:58:11.864604 21750 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:11.865432 21750 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:11.866535 21746 ts_tablet_manager.cc:1428] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.056s user 0.003s sys 0.008s
I20250114 08:58:11.867534 21746 tablet_bootstrap.cc:492] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:11.867890 21750 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:11.868932 21686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 001d4f9f4b5d4124b95dcddb6adac2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.869300 21611 tablet_service.cc:1467] Processing CreateTablet for tablet abedb4afaad04af9bbbb7d4266fd2632 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250114 08:58:11.870610 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abedb4afaad04af9bbbb7d4266fd2632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.875732 21619 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250114 08:58:11.877110 21619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.879052 21694 tablet_service.cc:1467] Processing CreateTablet for tablet abedb4afaad04af9bbbb7d4266fd2632 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250114 08:58:11.880420 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abedb4afaad04af9bbbb7d4266fd2632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.888592 21618 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4913bb74ad4fab98dd737eb0bd5965 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250114 08:58:11.916340 21751 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:11.917117 21751 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:11.919682 21688 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250114 08:58:11.908675 21744 tablet_bootstrap.cc:492] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:11.925727 21538 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250114 08:58:11.926055 21687 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7e8860a1d145fbb31ef3dc7864f2d0 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250114 08:58:11.922932 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.930616 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4913bb74ad4fab98dd737eb0bd5965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.930977 21691 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5c7ffbe79e4145825848346fd7652c (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250114 08:58:11.930656 21751 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:11.933470 21689 tablet_service.cc:1467] Processing CreateTablet for tablet ada199404e61479593fa1b64c574bcb8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250114 08:58:11.935889 21690 tablet_service.cc:1467] Processing CreateTablet for tablet 3e70c3cb3e8e4656945ec18e88e46101 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250114 08:58:11.938174 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.962005 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5c7ffbe79e4145825848346fd7652c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.974053 21690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e70c3cb3e8e4656945ec18e88e46101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.975999 21687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7e8860a1d145fbb31ef3dc7864f2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.979584 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada199404e61479593fa1b64c574bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:11.994639 21746 tablet_bootstrap.cc:654] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:11.994642 21745 tablet_bootstrap.cc:492] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:12.010629 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7e8860a1d145fbb31ef3dc7864f2d0 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250114 08:58:12.011633 21745 ts_tablet_manager.cc:1397] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.178s user 0.012s sys 0.033s
I20250114 08:58:12.012229 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7e8860a1d145fbb31ef3dc7864f2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.014634 21745 raft_consensus.cc:357] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.015419 21745 raft_consensus.cc:383] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.015717 21745 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.015973 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:12.016790 21744 tablet_bootstrap.cc:654] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:12.017993 21521 leader_election.cc:343] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.026350 21692 tablet_service.cc:1467] Processing CreateTablet for tablet ee02341e5e574d55aac7eb9b3df9dc73 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250114 08:58:12.046789 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5c7ffbe79e4145825848346fd7652c (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250114 08:58:12.047235 21685 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77833a006f478e86764142c505ab6b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250114 08:58:12.048223 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5c7ffbe79e4145825848346fd7652c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.068688 21744 tablet_bootstrap.cc:492] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:12.069267 21744 ts_tablet_manager.cc:1397] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.165s user 0.010s sys 0.032s
I20250114 08:58:12.072613 21692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee02341e5e574d55aac7eb9b3df9dc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.072286 21744 raft_consensus.cc:357] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.073220 21744 raft_consensus.cc:383] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.073519 21744 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.074393 21744 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.076813 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e77833a006f478e86764142c505ab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.076946 21744 ts_tablet_manager.cc:1428] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250114 08:58:12.079468 21615 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7e8860a1d145fbb31ef3dc7864f2d0 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250114 08:58:12.089491 21613 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5c7ffbe79e4145825848346fd7652c (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250114 08:58:12.089672 21616 tablet_service.cc:1467] Processing CreateTablet for tablet ada199404e61479593fa1b64c574bcb8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250114 08:58:12.091519 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 3e70c3cb3e8e4656945ec18e88e46101 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250114 08:58:12.093981 21539 tablet_service.cc:1467] Processing CreateTablet for tablet ada199404e61479593fa1b64c574bcb8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250114 08:58:12.094168 21612 tablet_service.cc:1467] Processing CreateTablet for tablet ee02341e5e574d55aac7eb9b3df9dc73 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250114 08:58:12.095268 21539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada199404e61479593fa1b64c574bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.082325 21744 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:12.096182 21694 tablet_service.cc:1467] Processing CreateTablet for tablet 92fb0c7bd28441aabcf6344f04b84ee6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250114 08:58:12.098914 21620 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77833a006f478e86764142c505ab6b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250114 08:58:12.101884 21617 tablet_service.cc:1467] Processing CreateTablet for tablet 92fb0c7bd28441aabcf6344f04b84ee6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250114 08:58:12.021909 21745 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.105845 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7e8860a1d145fbb31ef3dc7864f2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.104593 21750 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.108764 21750 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.109933 21620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e77833a006f478e86764142c505ab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.112731 21612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee02341e5e574d55aac7eb9b3df9dc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.119030 21617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb0c7bd28441aabcf6344f04b84ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.119678 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e70c3cb3e8e4656945ec18e88e46101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.120935 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada199404e61479593fa1b64c574bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.122745 21619 tablet_service.cc:1467] Processing CreateTablet for tablet cc4174b5896143c7b50c7abfd618a899 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250114 08:58:12.123766 21613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5c7ffbe79e4145825848346fd7652c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.131762 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 3e70c3cb3e8e4656945ec18e88e46101 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250114 08:58:12.133697 21752 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.134729 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e70c3cb3e8e4656945ec18e88e46101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.134238 21752 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:12.145390 21542 tablet_service.cc:1467] Processing CreateTablet for tablet ee02341e5e574d55aac7eb9b3df9dc73 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250114 08:58:12.146770 21542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee02341e5e574d55aac7eb9b3df9dc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.149811 21752 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:12.158852 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.142982 21750 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:12.159631 21555 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:12.159835 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6879e06c934d438e3ca9c3f093d4be (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250114 08:58:12.163645 21594 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:12.172053 21744 tablet_bootstrap.cc:654] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.172709 21751 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:12.173189 21746 tablet_bootstrap.cc:492] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:12.173084 21751 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:12.163136 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb0c7bd28441aabcf6344f04b84ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.175307 21746 ts_tablet_manager.cc:1397] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.308s user 0.033s sys 0.011s
I20250114 08:58:12.189251 21751 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:12.164599 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:12.192993 21545 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77833a006f478e86764142c505ab6b (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20250114 08:58:12.193598 21521 leader_election.cc:343] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.162413 21538 tablet_service.cc:1467] Processing CreateTablet for tablet cc4174b5896143c7b50c7abfd618a899 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250114 08:58:12.194980 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e77833a006f478e86764142c505ab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.174925 21619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc4174b5896143c7b50c7abfd618a899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.208389 21686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6879e06c934d438e3ca9c3f093d4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.208719 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc4174b5896143c7b50c7abfd618a899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.216224 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 3b369b35736c413ca069db061ed36bc5 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250114 08:58:12.216831 21543 tablet_service.cc:1467] Processing CreateTablet for tablet 3b369b35736c413ca069db061ed36bc5 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250114 08:58:12.217622 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b369b35736c413ca069db061ed36bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.221040 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 92fb0c7bd28441aabcf6344f04b84ee6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250114 08:58:12.160456 21691 tablet_service.cc:1467] Processing CreateTablet for tablet cc4174b5896143c7b50c7abfd618a899 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250114 08:58:12.176088 21536 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6879e06c934d438e3ca9c3f093d4be (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250114 08:58:12.229663 21746 raft_consensus.cc:357] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.230587 21746 raft_consensus.cc:383] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.230933 21746 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.232795 21618 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6879e06c934d438e3ca9c3f093d4be (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250114 08:58:12.234184 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6879e06c934d438e3ca9c3f093d4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.236860 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:12.238400 21596 leader_election.cc:343] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.231788 21746 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.247577 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:12.249104 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:12.252893 21543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b369b35736c413ca069db061ed36bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.253576 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb0c7bd28441aabcf6344f04b84ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.256755 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6879e06c934d438e3ca9c3f093d4be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:12.252354 21520 leader_election.cc:343] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.264899 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:12.266274 21750 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:12.282200 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:12.283811 21669 leader_election.cc:343] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.292654 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc4174b5896143c7b50c7abfd618a899. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:12.299376 21439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 69950.5us
I20250114 08:58:12.299844 21692 tablet_service.cc:1467] Processing CreateTablet for tablet 3b369b35736c413ca069db061ed36bc5 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250114 08:58:12.300786 21752 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:12.301398 21438 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 58141us
I20250114 08:58:12.301268 21752 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.302511 21689 tablet_service.cc:1467] Processing CreateTablet for tablet cc5506492f514407a5097b5b2e8c7042 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250114 08:58:12.303375 21752 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:12.304031 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5506492f514407a5097b5b2e8c7042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.312728 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.312951 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:12.313172 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.313534 21630 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:12.313481 21617 tablet_service.cc:1467] Processing CreateTablet for tablet cc5506492f514407a5097b5b2e8c7042 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250114 08:58:12.315182 21669 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:12.315447 21537 tablet_service.cc:1467] Processing CreateTablet for tablet cc5506492f514407a5097b5b2e8c7042 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250114 08:58:12.316051 21752 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:12.316426 21752 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:12.316579 21617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5506492f514407a5097b5b2e8c7042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.316838 21752 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:12.316696 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5506492f514407a5097b5b2e8c7042. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:12.315630 21668 leader_election.cc:343] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.316660 21745 ts_tablet_manager.cc:1428] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.305s user 0.015s sys 0.011s
I20250114 08:58:12.321318 21692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b369b35736c413ca069db061ed36bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.327121 21745 tablet_bootstrap.cc:492] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:12.326519 21744 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:12.331514 21744 ts_tablet_manager.cc:1397] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.254s user 0.017s sys 0.009s
I20250114 08:58:12.334605 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 9690a0a2cde84c5d9312c8694185a0b2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250114 08:58:12.335090 21690 tablet_service.cc:1467] Processing CreateTablet for tablet 9690a0a2cde84c5d9312c8694185a0b2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250114 08:58:12.339344 21769 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.323421 21668 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:12.343456 21690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9690a0a2cde84c5d9312c8694185a0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.343021 21769 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.343582 21685 tablet_service.cc:1467] Processing CreateTablet for tablet be5a427c5ff04f9fa27e3e64bd27b146 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250114 08:58:12.344951 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5a427c5ff04f9fa27e3e64bd27b146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.346781 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.347921 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:12.348239 21594 leader_election.cc:343] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.348786 21769 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:12.351027 21746 ts_tablet_manager.cc:1428] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.159s user 0.013s sys 0.005s
I20250114 08:58:12.352068 21746 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:12.353358 21616 tablet_service.cc:1467] Processing CreateTablet for tablet be5a427c5ff04f9fa27e3e64bd27b146 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250114 08:58:12.356256 21542 tablet_service.cc:1467] Processing CreateTablet for tablet be5a427c5ff04f9fa27e3e64bd27b146 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250114 08:58:12.357622 21542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5a427c5ff04f9fa27e3e64bd27b146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.357749 21744 raft_consensus.cc:357] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.358553 21744 raft_consensus.cc:383] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.358851 21744 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.359560 21744 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.365902 21751 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.367200 21629 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:12.372864 21520 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:12.373852 21764 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:12.377290 21620 tablet_service.cc:1467] Processing CreateTablet for tablet d0b4ce684a6943809542800e14d94079 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250114 08:58:12.379505 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:12.379935 21752 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.379468 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9690a0a2cde84c5d9312c8694185a0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.381172 21704 raft_consensus.cc:2388] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:12.381477 21620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b4ce684a6943809542800e14d94079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.382215 21764 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:12.383044 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5a427c5ff04f9fa27e3e64bd27b146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.382969 21596 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:12.384238 21769 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:12.388701 21752 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:12.389295 21752 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:12.389398 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:12.391108 21745 tablet_bootstrap.cc:654] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.391719 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:12.393153 21596 leader_election.cc:343] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.394395 21751 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:12.394208 21538 tablet_service.cc:1467] Processing CreateTablet for tablet d0b4ce684a6943809542800e14d94079 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250114 08:58:12.395538 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b4ce684a6943809542800e14d94079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.397349 21545 tablet_service.cc:1467] Processing CreateTablet for tablet 9690a0a2cde84c5d9312c8694185a0b2 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250114 08:58:12.409129 21689 tablet_service.cc:1467] Processing CreateTablet for tablet d0b4ce684a6943809542800e14d94079 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250114 08:58:12.423967 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b4ce684a6943809542800e14d94079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.424798 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9690a0a2cde84c5d9312c8694185a0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.427069 21612 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba4257b18434159a50a4f6ae4a0d736 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250114 08:58:12.428921 21612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba4257b18434159a50a4f6ae4a0d736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.415987 21769 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:12.400431 21746 tablet_bootstrap.cc:654] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.401500 21554 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:12.440510 21594 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:12.421716 21543 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba4257b18434159a50a4f6ae4a0d736 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250114 08:58:12.442438 21543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba4257b18434159a50a4f6ae4a0d736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.418684 21752 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.444545 21691 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba4257b18434159a50a4f6ae4a0d736 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250114 08:58:12.444820 21752 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.445895 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba4257b18434159a50a4f6ae4a0d736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.446905 21752 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:12.452530 21629 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:12.456903 21669 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:12.459379 21769 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:12.461586 21752 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:12.462826 21769 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:12.462884 21752 raft_consensus.cc:695] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:12.463788 21769 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.464972 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 6846f3a6d80748369341fe169ebc427e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250114 08:58:12.465145 21688 tablet_service.cc:1467] Processing CreateTablet for tablet 6846f3a6d80748369341fe169ebc427e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250114 08:58:12.466267 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846f3a6d80748369341fe169ebc427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.466380 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846f3a6d80748369341fe169ebc427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.471967 21615 tablet_service.cc:1467] Processing CreateTablet for tablet a14dbff33afe4ab483f01265cb35d676 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250114 08:58:12.471958 21544 tablet_service.cc:1467] Processing CreateTablet for tablet a14dbff33afe4ab483f01265cb35d676 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250114 08:58:12.473672 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a14dbff33afe4ab483f01265cb35d676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.477353 21536 tablet_service.cc:1467] Processing CreateTablet for tablet 6846f3a6d80748369341fe169ebc427e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
W20250114 08:58:12.469002 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 153527us
I20250114 08:58:12.482709 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a14dbff33afe4ab483f01265cb35d676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.484122 21744 ts_tablet_manager.cc:1428] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.152s user 0.020s sys 0.017s
I20250114 08:58:12.485212 21744 tablet_bootstrap.cc:492] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:12.486631 21745 tablet_bootstrap.cc:492] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:12.487124 21745 ts_tablet_manager.cc:1397] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.160s user 0.009s sys 0.004s
I20250114 08:58:12.481570 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846f3a6d80748369341fe169ebc427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.479260 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:12.498303 21669 leader_election.cc:343] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
W20250114 08:58:12.507992 21668 leader_election.cc:343] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.510669 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:12.530791 21617 tablet_service.cc:1467] Processing CreateTablet for tablet f466e3efd6cb41329852e9028490bdf7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250114 08:58:12.463793 21752 consensus_queue.cc:237] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.532195 21617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f466e3efd6cb41329852e9028490bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.533874 21745 raft_consensus.cc:357] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.534706 21745 raft_consensus.cc:383] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.535019 21745 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.535074 21691 tablet_service.cc:1467] Processing CreateTablet for tablet f466e3efd6cb41329852e9028490bdf7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250114 08:58:12.537796 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f466e3efd6cb41329852e9028490bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.506156 21685 tablet_service.cc:1467] Processing CreateTablet for tablet a14dbff33afe4ab483f01265cb35d676 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250114 08:58:12.540195 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a14dbff33afe4ab483f01265cb35d676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.551268 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.550652 21451 catalog_manager.cc:5526] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:12.553251 21539 tablet_service.cc:1467] Processing CreateTablet for tablet 106dea9d52554552a1b3e20e7c82a3a8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250114 08:58:12.553272 21542 tablet_service.cc:1467] Processing CreateTablet for tablet f466e3efd6cb41329852e9028490bdf7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250114 08:58:12.555892 21539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106dea9d52554552a1b3e20e7c82a3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.556779 21542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f466e3efd6cb41329852e9028490bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:12.560587 21668 leader_election.cc:343] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.549419 21745 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:12.561563 21668 leader_election.cc:343] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.561975 21668 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:12.577960 21771 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:12.579571 21620 tablet_service.cc:1467] Processing CreateTablet for tablet 106dea9d52554552a1b3e20e7c82a3a8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250114 08:58:12.580952 21620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106dea9d52554552a1b3e20e7c82a3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.585777 21688 tablet_service.cc:1467] Processing CreateTablet for tablet 106dea9d52554552a1b3e20e7c82a3a8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250114 08:58:12.587097 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106dea9d52554552a1b3e20e7c82a3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.587922 21745 ts_tablet_manager.cc:1428] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.100s user 0.015s sys 0.000s
I20250114 08:58:12.599572 21543 tablet_service.cc:1467] Processing CreateTablet for tablet 2d803a1f3141467fa3c2c63696526f84 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250114 08:58:12.599843 21618 tablet_service.cc:1467] Processing CreateTablet for tablet 2d803a1f3141467fa3c2c63696526f84 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250114 08:58:12.604094 21692 tablet_service.cc:1467] Processing CreateTablet for tablet 2d803a1f3141467fa3c2c63696526f84 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250114 08:58:12.605719 21692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d803a1f3141467fa3c2c63696526f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.606673 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d803a1f3141467fa3c2c63696526f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.607879 21538 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3a3fdbbbca423695a05eb175031e72 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250114 08:58:12.621942 21744 tablet_bootstrap.cc:654] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.622124 21746 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:12.622586 21746 ts_tablet_manager.cc:1397] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.271s user 0.005s sys 0.027s
I20250114 08:58:12.625023 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 0205fa1ad333433a8a509e5ad9620de8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250114 08:58:12.625918 21690 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3a3fdbbbca423695a05eb175031e72 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250114 08:58:12.627233 21690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c3a3fdbbbca423695a05eb175031e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.626330 21746 raft_consensus.cc:357] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.617677 21543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d803a1f3141467fa3c2c63696526f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.632867 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0205fa1ad333433a8a509e5ad9620de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.633581 21616 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3a3fdbbbca423695a05eb175031e72 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250114 08:58:12.634958 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c3a3fdbbbca423695a05eb175031e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.631790 21745 tablet_bootstrap.cc:492] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:12.640897 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c3a3fdbbbca423695a05eb175031e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.628515 21746 raft_consensus.cc:383] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.645079 21746 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
W20250114 08:58:12.646684 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:12.646369 21746 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.650014 21746 ts_tablet_manager.cc:1428] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20250114 08:58:12.661902 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 0205fa1ad333433a8a509e5ad9620de8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250114 08:58:12.663358 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0205fa1ad333433a8a509e5ad9620de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.666482 21540 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2d1d6b0f14582b7864c1b80a47f26 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250114 08:58:12.668356 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e2d1d6b0f14582b7864c1b80a47f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.682637 21689 tablet_service.cc:1467] Processing CreateTablet for tablet 0205fa1ad333433a8a509e5ad9620de8 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250114 08:58:12.689565 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0205fa1ad333433a8a509e5ad9620de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.636283 21452 catalog_manager.cc:5526] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:12.687150 21686 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2d1d6b0f14582b7864c1b80a47f26 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250114 08:58:12.683588 21615 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2d1d6b0f14582b7864c1b80a47f26 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250114 08:58:12.705760 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e2d1d6b0f14582b7864c1b80a47f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.705955 21686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e2d1d6b0f14582b7864c1b80a47f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.713230 21746 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:12.695987 21745 tablet_bootstrap.cc:654] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.723645 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa5b3173173431d91e459f9fdcb61a9 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250114 08:58:12.725014 21611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa5b3173173431d91e459f9fdcb61a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.700143 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa5b3173173431d91e459f9fdcb61a9 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250114 08:58:12.730271 21691 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa5b3173173431d91e459f9fdcb61a9 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250114 08:58:12.730763 21694 tablet_service.cc:1467] Processing CreateTablet for tablet eb00de7db5024821adbfdfd3f8a0f475 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250114 08:58:12.731127 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa5b3173173431d91e459f9fdcb61a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.731603 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa5b3173173431d91e459f9fdcb61a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.731948 21617 tablet_service.cc:1467] Processing CreateTablet for tablet eb00de7db5024821adbfdfd3f8a0f475 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250114 08:58:12.733312 21617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb00de7db5024821adbfdfd3f8a0f475. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.737929 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb00de7db5024821adbfdfd3f8a0f475. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.755939 21538 tablet_service.cc:1467] Processing CreateTablet for tablet 270ce1b254bd4b7180e48e18af79cda6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250114 08:58:12.757465 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 270ce1b254bd4b7180e48e18af79cda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.772770 21542 tablet_service.cc:1467] Processing CreateTablet for tablet eb00de7db5024821adbfdfd3f8a0f475 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250114 08:58:12.784583 21542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb00de7db5024821adbfdfd3f8a0f475. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.801231 21616 tablet_service.cc:1467] Processing CreateTablet for tablet 270ce1b254bd4b7180e48e18af79cda6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250114 08:58:12.802393 21744 tablet_bootstrap.cc:492] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:12.802624 21616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 270ce1b254bd4b7180e48e18af79cda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.803049 21744 ts_tablet_manager.cc:1397] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.318s user 0.053s sys 0.078s
I20250114 08:58:12.809413 21690 tablet_service.cc:1467] Processing CreateTablet for tablet 270ce1b254bd4b7180e48e18af79cda6 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250114 08:58:12.813355 21744 raft_consensus.cc:357] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.814163 21744 raft_consensus.cc:383] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.814460 21744 raft_consensus.cc:738] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.815416 21744 consensus_queue.cc:260] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.818631 21690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 270ce1b254bd4b7180e48e18af79cda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.820096 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 82151d61cef04050b69f24bc614507fc (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250114 08:58:12.821477 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82151d61cef04050b69f24bc614507fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.820096 21537 tablet_service.cc:1467] Processing CreateTablet for tablet f21ba5689065467b967a5802582959dd (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250114 08:58:12.827586 21537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21ba5689065467b967a5802582959dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.828637 21687 tablet_service.cc:1467] Processing CreateTablet for tablet 82151d61cef04050b69f24bc614507fc (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250114 08:58:12.829886 21687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82151d61cef04050b69f24bc614507fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.830952 21688 tablet_service.cc:1467] Processing CreateTablet for tablet f21ba5689065467b967a5802582959dd (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250114 08:58:12.835548 21752 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.836035 21752 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.838711 21746 tablet_bootstrap.cc:654] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.839684 21612 tablet_service.cc:1467] Processing CreateTablet for tablet f21ba5689065467b967a5802582959dd (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250114 08:58:12.841419 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:12.839737 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.842329 21752 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:12.845607 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 4c244f79365347c49406af3a7a5a55a7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20250114 08:58:12.849260 21669 leader_election.cc:343] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.849757 21693 tablet_service.cc:1467] Processing CreateTablet for tablet 4c244f79365347c49406af3a7a5a55a7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20250114 08:58:12.850407 21668 leader_election.cc:343] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.850865 21668 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:12.851689 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c244f79365347c49406af3a7a5a55a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.853626 21752 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:12.856534 21543 tablet_service.cc:1467] Processing CreateTablet for tablet fec3bfdda69d4934970f1e752db88d50 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250114 08:58:12.857861 21543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec3bfdda69d4934970f1e752db88d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.842140 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 82151d61cef04050b69f24bc614507fc (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250114 08:58:12.861840 21544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82151d61cef04050b69f24bc614507fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.865299 21769 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:12.867048 21539 tablet_service.cc:1467] Processing CreateTablet for tablet baef410f99d34fe9a53080abafb2259e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 08:58:12.868364 21539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baef410f99d34fe9a53080abafb2259e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.844410 21745 tablet_bootstrap.cc:492] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:12.875208 21612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21ba5689065467b967a5802582959dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.875818 21536 tablet_service.cc:1467] Processing CreateTablet for tablet 3e648ebc4e4d4a4b9a35eefeb989e0fe (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250114 08:58:12.877542 21545 tablet_service.cc:1467] Processing CreateTablet for tablet b2d157d78e1e4854b971c0382cb7fe21 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250114 08:58:12.878329 21745 ts_tablet_manager.cc:1397] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.262s user 0.013s sys 0.008s
I20250114 08:58:12.887152 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e648ebc4e4d4a4b9a35eefeb989e0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.887929 21691 tablet_service.cc:1467] Processing CreateTablet for tablet fec3bfdda69d4934970f1e752db88d50 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250114 08:58:12.888422 21686 tablet_service.cc:1467] Processing CreateTablet for tablet baef410f99d34fe9a53080abafb2259e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 08:58:12.889030 21745 raft_consensus.cc:357] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:12.889590 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d157d78e1e4854b971c0382cb7fe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.889824 21745 raft_consensus.cc:383] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.890165 21745 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.890934 21764 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.891371 21764 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.893496 21764 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:12.895116 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:12.897284 21745 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:12.898252 21694 tablet_service.cc:1467] Processing CreateTablet for tablet 3e648ebc4e4d4a4b9a35eefeb989e0fe (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250114 08:58:12.899928 21745 ts_tablet_manager.cc:1428] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20250114 08:58:12.903764 21688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21ba5689065467b967a5802582959dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.920116 21617 tablet_service.cc:1467] Processing CreateTablet for tablet baef410f99d34fe9a53080abafb2259e (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 08:58:12.921510 21617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baef410f99d34fe9a53080abafb2259e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.924783 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec3bfdda69d4934970f1e752db88d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.925457 21614 tablet_service.cc:1467] Processing CreateTablet for tablet fec3bfdda69d4934970f1e752db88d50 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250114 08:58:12.905941 21744 ts_tablet_manager.cc:1428] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.102s user 0.010s sys 0.001s
I20250114 08:58:12.928789 21693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c244f79365347c49406af3a7a5a55a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.930140 21746 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:12.930701 21746 ts_tablet_manager.cc:1397] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.218s user 0.016s sys 0.007s
I20250114 08:58:12.932518 21686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baef410f99d34fe9a53080abafb2259e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.915324 21620 tablet_service.cc:1467] Processing CreateTablet for tablet 4c244f79365347c49406af3a7a5a55a7 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250114 08:58:12.917902 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:12.934587 21629 raft_consensus.cc:2388] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:12.934456 21618 tablet_service.cc:1467] Processing CreateTablet for tablet b2d157d78e1e4854b971c0382cb7fe21 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250114 08:58:12.925457 21615 tablet_service.cc:1467] Processing CreateTablet for tablet 3e648ebc4e4d4a4b9a35eefeb989e0fe (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250114 08:58:12.934818 21685 tablet_service.cc:1467] Processing CreateTablet for tablet b2d157d78e1e4854b971c0382cb7fe21 (DEFAULT_TABLE table=test-workload [id=c01c36aed5b24598bdaa599e3312866e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250114 08:58:12.937122 21685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d157d78e1e4854b971c0382cb7fe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.938429 21744 tablet_bootstrap.cc:492] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:12.940625 21751 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.941102 21751 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.943130 21751 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:12.945008 21555 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:12.948290 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:12.948740 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e648ebc4e4d4a4b9a35eefeb989e0fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:12.954138 21594 leader_election.cc:343] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:12.955423 21520 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:12.958135 21746 raft_consensus.cc:357] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.959059 21746 raft_consensus.cc:383] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:12.959426 21746 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:12.960325 21746 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:12.949770 21745 tablet_bootstrap.cc:492] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:12.971756 21764 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 08:58:12.978708 21596 consensus_peers.cc:487] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed -> Peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Couldn't send request to peer cec0ef3bbb654988bf2123d8eef7a0a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:12.978689 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:12.979707 21704 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:12.982437 21596 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:12.983069 21764 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:12.983767 21745 tablet_bootstrap.cc:654] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:12.983646 21764 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:12.975796 21614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec3bfdda69d4934970f1e752db88d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:12.989496 21751 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:12.995549 21615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e648ebc4e4d4a4b9a35eefeb989e0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:13.001082 21752 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:13.005095 21744 tablet_bootstrap.cc:654] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.007375 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:12.995761 21751 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.008687 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d157d78e1e4854b971c0382cb7fe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:13.009003 21751 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.009614 21620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c244f79365347c49406af3a7a5a55a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 08:58:13.023216 21520 leader_election.cc:343] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.033200 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.017900 21764 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.034014 21704 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.035411 21521 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.036594 21764 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.036959 21764 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.037323 21764 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.038478 21746 ts_tablet_manager.cc:1428] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.107s user 0.007s sys 0.000s
I20250114 08:58:13.044776 21764 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:13.063148 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.063804 21704 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.071615 21746 tablet_bootstrap.cc:492] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.060900 21764 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.080739 21704 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.082268 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:13.083278 21521 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:13.083679 21744 tablet_bootstrap.cc:492] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.084132 21744 ts_tablet_manager.cc:1397] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.146s user 0.021s sys 0.028s
I20250114 08:58:13.084687 21764 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
W20250114 08:58:13.084121 21520 leader_election.cc:343] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.086457 21764 raft_consensus.cc:695] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.087178 21744 raft_consensus.cc:357] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.087971 21744 raft_consensus.cc:383] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.087365 21764 consensus_queue.cc:237] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:13.088267 21744 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.089278 21744 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:13.050822 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 50219.1us
I20250114 08:58:13.095641 21751 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.098546 21751 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.105551 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.106271 21704 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.108825 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:13.106194 21744 ts_tablet_manager.cc:1428] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250114 08:58:13.115973 21744 tablet_bootstrap.cc:492] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
W20250114 08:58:13.126680 21594 leader_election.cc:343] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.127619 21745 tablet_bootstrap.cc:492] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.128142 21745 ts_tablet_manager.cc:1397] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.226s user 0.013s sys 0.006s
I20250114 08:58:13.129987 21746 tablet_bootstrap.cc:654] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.135792 21704 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:13.137611 21745 raft_consensus.cc:357] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.138299 21745 raft_consensus.cc:383] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.138597 21745 raft_consensus.cc:738] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.139196 21745 consensus_queue.cc:260] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.140807 21596 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:13.141695 21751 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.142380 21751 raft_consensus.cc:695] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:13.142678 21745 ts_tablet_manager.cc:1428] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20250114 08:58:13.145295 21744 tablet_bootstrap.cc:654] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.143548 21751 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.143563 21745 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.159279 21746 tablet_bootstrap.cc:492] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.159976 21746 ts_tablet_manager.cc:1397] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.016s
I20250114 08:58:13.161535 21451 catalog_manager.cc:5526] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:13.162940 21746 raft_consensus.cc:357] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.163767 21746 raft_consensus.cc:383] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.164083 21746 raft_consensus.cc:738] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.166556 21745 tablet_bootstrap.cc:654] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.168584 21752 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:13.164999 21746 consensus_queue.cc:260] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.176504 21746 ts_tablet_manager.cc:1428] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.016s user 0.009s sys 0.001s
I20250114 08:58:13.177814 21746 tablet_bootstrap.cc:492] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.188019 21452 catalog_manager.cc:5526] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.201753 21746 tablet_bootstrap.cc:654] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.202210 21745 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.202745 21745 ts_tablet_manager.cc:1397] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.011s
I20250114 08:58:13.206636 21744 tablet_bootstrap.cc:492] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.206030 21745 raft_consensus.cc:357] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.207151 21744 ts_tablet_manager.cc:1397] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.091s user 0.016s sys 0.004s
I20250114 08:58:13.207120 21745 raft_consensus.cc:383] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.207756 21745 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.208735 21745 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.210952 21744 raft_consensus.cc:357] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.211699 21744 raft_consensus.cc:383] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.211984 21745 ts_tablet_manager.cc:1428] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250114 08:58:13.211983 21744 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.213084 21745 tablet_bootstrap.cc:492] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.213258 21744 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.215610 21744 ts_tablet_manager.cc:1428] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20250114 08:58:13.218250 21746 tablet_bootstrap.cc:492] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.218739 21746 ts_tablet_manager.cc:1397] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.006s
I20250114 08:58:13.219208 21745 tablet_bootstrap.cc:654] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.219933 21798 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.220487 21798 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.221244 21744 tablet_bootstrap.cc:492] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.224371 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:13.226339 21669 leader_election.cc:343] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.227696 21744 tablet_bootstrap.cc:654] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.228824 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.229265 21746 raft_consensus.cc:357] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:13.229718 21554 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:13.230075 21746 raft_consensus.cc:383] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.230434 21746 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.231200 21746 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:13.231721 21668 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.233254 21771 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.233749 21746 ts_tablet_manager.cc:1428] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250114 08:58:13.233700 21771 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.234282 21771 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.234839 21746 tablet_bootstrap.cc:492] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.242269 21798 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:13.248332 21744 tablet_bootstrap.cc:492] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.248878 21744 ts_tablet_manager.cc:1397] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.009s
I20250114 08:58:13.242812 21746 tablet_bootstrap.cc:654] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.251119 21771 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.251772 21744 raft_consensus.cc:357] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.252806 21744 raft_consensus.cc:383] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.253214 21744 raft_consensus.cc:738] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.254168 21744 consensus_queue.cc:260] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.255290 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:13.256057 21554 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.257203 21744 ts_tablet_manager.cc:1428] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250114 08:58:13.258852 21744 tablet_bootstrap.cc:492] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.259054 21745 tablet_bootstrap.cc:492] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.259505 21745 ts_tablet_manager.cc:1397] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.047s user 0.023s sys 0.019s
I20250114 08:58:13.259999 21771 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:13.261440 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:13.262353 21745 raft_consensus.cc:357] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.263149 21745 raft_consensus.cc:383] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.263453 21745 raft_consensus.cc:738] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.264137 21745 consensus_queue.cc:260] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:13.265152 21669 leader_election.cc:343] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.266709 21746 tablet_bootstrap.cc:492] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.266745 21745 ts_tablet_manager.cc:1428] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250114 08:58:13.267411 21746 ts_tablet_manager.cc:1397] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.005s
I20250114 08:58:13.268191 21745 tablet_bootstrap.cc:492] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.265134 21744 tablet_bootstrap.cc:654] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.270614 21746 raft_consensus.cc:357] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.271484 21746 raft_consensus.cc:383] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.271899 21746 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.272781 21746 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.276015 21745 tablet_bootstrap.cc:654] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.278589 21746 ts_tablet_manager.cc:1428] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20250114 08:58:13.279619 21746 tablet_bootstrap.cc:492] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.282629 21744 tablet_bootstrap.cc:492] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.283105 21744 ts_tablet_manager.cc:1397] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.001s
I20250114 08:58:13.284590 21745 tablet_bootstrap.cc:492] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.284992 21745 ts_tablet_manager.cc:1397] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20250114 08:58:13.285615 21744 raft_consensus.cc:357] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.286314 21744 raft_consensus.cc:383] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.286589 21744 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.287320 21745 raft_consensus.cc:357] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.287297 21744 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.288070 21745 raft_consensus.cc:383] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.288374 21745 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.288861 21746 tablet_bootstrap.cc:654] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.288977 21745 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.289928 21744 ts_tablet_manager.cc:1428] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250114 08:58:13.290879 21744 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.291427 21745 ts_tablet_manager.cc:1428] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 08:58:13.292279 21745 tablet_bootstrap.cc:492] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.296286 21744 tablet_bootstrap.cc:654] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.297561 21745 tablet_bootstrap.cc:654] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.304049 21746 tablet_bootstrap.cc:492] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.304754 21746 ts_tablet_manager.cc:1397] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.010s
I20250114 08:58:13.307453 21746 raft_consensus.cc:357] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.303977 21554 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:13.309011 21745 tablet_bootstrap.cc:492] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.308377 21746 raft_consensus.cc:383] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.309238 21668 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.309682 21745 ts_tablet_manager.cc:1397] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20250114 08:58:13.309700 21746 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.310158 21771 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.310688 21771 raft_consensus.cc:695] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:13.310775 21746 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.311575 21771 consensus_queue.cc:237] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.312441 21745 raft_consensus.cc:357] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.313165 21745 raft_consensus.cc:383] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.313428 21745 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.314060 21746 ts_tablet_manager.cc:1428] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 08:58:13.314039 21745 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.314630 21751 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.315034 21751 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.316576 21745 ts_tablet_manager.cc:1428] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250114 08:58:13.318105 21745 tablet_bootstrap.cc:492] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.318212 21746 tablet_bootstrap.cc:492] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.317431 21751 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.322387 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.323143 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:13.324458 21594 leader_election.cc:343] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.327633 21744 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.328136 21744 ts_tablet_manager.cc:1397] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.012s
I20250114 08:58:13.329522 21745 tablet_bootstrap.cc:654] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:13.330907 21596 leader_election.cc:343] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.331332 21596 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.331915 21751 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.330907 21744 raft_consensus.cc:357] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.333936 21746 tablet_bootstrap.cc:654] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.334002 21744 raft_consensus.cc:383] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.335577 21744 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.336244 21744 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.337671 21751 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.338122 21751 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.338721 21744 ts_tablet_manager.cc:1428] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250114 08:58:13.339581 21744 tablet_bootstrap.cc:492] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.340114 21751 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.341925 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.342402 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.343106 21554 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
W20250114 08:58:13.343961 21596 leader_election.cc:343] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.344806 21594 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.336503 21450 catalog_manager.cc:5526] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:13.346271 21751 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.346521 21744 tablet_bootstrap.cc:654] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.348591 21751 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.349119 21751 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.353168 21745 tablet_bootstrap.cc:492] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.353741 21745 ts_tablet_manager.cc:1397] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.002s
I20250114 08:58:13.356284 21745 raft_consensus.cc:357] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.357376 21745 raft_consensus.cc:383] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.357816 21745 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.358702 21745 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.361620 21745 ts_tablet_manager.cc:1428] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250114 08:58:13.365170 21745 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.364182 21751 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.369549 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:13.370406 21554 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.372617 21764 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.372870 21746 tablet_bootstrap.cc:492] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.373490 21746 ts_tablet_manager.cc:1397] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.021s
I20250114 08:58:13.373142 21764 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.375262 21764 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.376209 21746 raft_consensus.cc:357] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.377020 21746 raft_consensus.cc:383] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.376860 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.377569 21704 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.378448 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.378793 21521 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:13.379760 21764 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.380179 21764 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.380666 21764 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.380857 21745 tablet_bootstrap.cc:654] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:13.380008 21520 leader_election.cc:343] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.381330 21554 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:13.382413 21594 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.383266 21769 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.383746 21769 raft_consensus.cc:695] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:13.383936 21791 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.384629 21769 consensus_queue.cc:237] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.384559 21791 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.387408 21791 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.388168 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.390820 21751 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.390832 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:13.391110 21520 leader_election.cc:343] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
W20250114 08:58:13.392985 21596 leader_election.cc:343] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.391074 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.394109 21703 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.395476 21521 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.395828 21745 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.396360 21745 ts_tablet_manager.cc:1397] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.004s
I20250114 08:58:13.396991 21744 tablet_bootstrap.cc:492] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.396912 21791 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.397416 21791 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.397512 21744 ts_tablet_manager.cc:1397] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.017s
I20250114 08:58:13.397758 21791 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.398916 21745 raft_consensus.cc:357] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.399618 21745 raft_consensus.cc:383] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.399866 21745 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.400123 21744 raft_consensus.cc:357] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.400853 21744 raft_consensus.cc:383] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.400570 21745 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.401104 21744 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.401911 21744 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.406603 21745 ts_tablet_manager.cc:1428] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250114 08:58:13.406946 21746 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.407528 21745 tablet_bootstrap.cc:492] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.409199 21744 ts_tablet_manager.cc:1428] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250114 08:58:13.410593 21791 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.413560 21744 tablet_bootstrap.cc:492] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.414300 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:13.415900 21520 leader_election.cc:343] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.411538 21746 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.409466 21764 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.419171 21746 ts_tablet_manager.cc:1428] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20250114 08:58:13.419358 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:13.425308 21746 tablet_bootstrap.cc:492] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
W20250114 08:58:13.425670 21520 leader_election.cc:343] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.420286 21764 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.421409 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.426442 21703 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.422453 21745 tablet_bootstrap.cc:654] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.423161 21744 tablet_bootstrap.cc:654] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.423117 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.429809 21704 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.432809 21746 tablet_bootstrap.cc:654] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.431610 21450 catalog_manager.cc:5526] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.435075 21769 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.435559 21769 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.441588 21703 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.449930 21521 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.451418 21786 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.452080 21786 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.455302 21786 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.456041 21786 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.458199 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.459182 21629 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.460876 21520 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.423856 21791 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.462980 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.465617 21769 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.465888 21554 raft_consensus.cc:2388] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:13.465746 21764 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.466379 21764 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.467207 21764 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.464686 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.468429 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.470731 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.471035 21702 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.471889 21769 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.448192 21704 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.475994 21521 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.476498 21745 tablet_bootstrap.cc:492] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.477120 21745 ts_tablet_manager.cc:1397] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.030s
I20250114 08:58:13.477231 21791 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.477878 21791 raft_consensus.cc:695] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.478803 21791 consensus_queue.cc:237] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.480779 21745 raft_consensus.cc:357] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.481653 21745 raft_consensus.cc:383] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.481105 21786 raft_consensus.cc:695] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.481954 21745 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.482640 21744 tablet_bootstrap.cc:492] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.483265 21744 ts_tablet_manager.cc:1397] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.070s user 0.016s sys 0.012s
I20250114 08:58:13.482733 21745 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.482645 21786 consensus_queue.cc:237] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.485164 21745 ts_tablet_manager.cc:1428] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20250114 08:58:13.486050 21745 tablet_bootstrap.cc:492] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.486131 21746 tablet_bootstrap.cc:492] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.486307 21744 raft_consensus.cc:357] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.486564 21746 ts_tablet_manager.cc:1397] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.020s
I20250114 08:58:13.486845 21744 raft_consensus.cc:383] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.487169 21744 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.483777 21769 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.487903 21744 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.490651 21744 ts_tablet_manager.cc:1428] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 08:58:13.491602 21744 tablet_bootstrap.cc:492] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.492656 21745 tablet_bootstrap.cc:654] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.490365 21746 raft_consensus.cc:357] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.494542 21746 raft_consensus.cc:383] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.494793 21746 raft_consensus.cc:738] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.495257 21746 consensus_queue.cc:260] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.496948 21764 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.500147 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:13.500524 21746 ts_tablet_manager.cc:1428] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20250114 08:58:13.501921 21746 tablet_bootstrap.cc:492] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.501001 21629 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.504873 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.502415 21764 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.509860 21746 tablet_bootstrap.cc:654] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.510376 21704 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.510257 21791 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.521942 21786 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.522539 21786 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.527204 21786 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.528537 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.531198 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.525374 21453 catalog_manager.cc:5526] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.521049 21791 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.534796 21702 raft_consensus.cc:2388] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:13.536293 21791 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.538826 21520 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.541373 21702 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.542452 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.543381 21630 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.550722 21744 tablet_bootstrap.cc:654] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.551108 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.552505 21629 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.553258 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.553752 21701 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.554440 21786 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.554831 21786 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.551712 21791 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.555604 21769 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.556088 21769 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:13.555202 21786 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.559239 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.562127 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.562973 21701 raft_consensus.cc:2388] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:13.564569 21704 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.564249 21769 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:13.565582 21596 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.566421 21769 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.566644 21520 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.568240 21751 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.569067 21764 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.568782 21751 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.569548 21764 raft_consensus.cc:695] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.570444 21764 consensus_queue.cc:237] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.571082 21751 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.573791 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.583437 21769 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.571058 21786 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.585891 21791 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.590839 21786 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.592303 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:13.593465 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.593832 21629 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.594071 21703 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.594816 21786 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:13.595803 21744 tablet_bootstrap.cc:492] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.595947 21746 tablet_bootstrap.cc:492] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.596403 21744 ts_tablet_manager.cc:1397] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.105s user 0.022s sys 0.001s
I20250114 08:58:13.596433 21746 ts_tablet_manager.cc:1397] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.095s user 0.023s sys 0.017s
I20250114 08:58:13.600996 21744 raft_consensus.cc:357] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.601995 21744 raft_consensus.cc:383] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.602353 21744 raft_consensus.cc:738] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.607373 21629 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.609143 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.610491 21786 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.599054 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.611140 21786 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.612053 21704 raft_consensus.cc:2388] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:13.612288 21744 consensus_queue.cc:260] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.612876 21786 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.613236 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.614789 21769 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.617070 21745 tablet_bootstrap.cc:492] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.617157 21744 ts_tablet_manager.cc:1428] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250114 08:58:13.617590 21745 ts_tablet_manager.cc:1397] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.132s user 0.028s sys 0.014s
I20250114 08:58:13.618036 21744 tablet_bootstrap.cc:492] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.613221 21703 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.621057 21746 raft_consensus.cc:357] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.621990 21746 raft_consensus.cc:383] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.622340 21746 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.610034 21450 catalog_manager.cc:5526] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.623131 21746 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.628224 21745 raft_consensus.cc:357] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.628911 21745 raft_consensus.cc:383] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.629189 21745 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.630128 21769 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.631438 21745 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.634716 21745 ts_tablet_manager.cc:1428] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250114 08:58:13.635320 21746 ts_tablet_manager.cc:1428] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.037s user 0.006s sys 0.001s
I20250114 08:58:13.635603 21745 tablet_bootstrap.cc:492] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.636206 21746 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
W20250114 08:58:13.654122 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:13.676261 21798 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.677031 21798 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.679073 21798 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:13.700820 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.706212 21786 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250114 08:58:13.710299 21554 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:13.709643 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.712247 21769 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:13.714802 21786 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.715312 21786 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.717298 21786 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.727970 21744 tablet_bootstrap.cc:654] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.725279 21669 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.728569 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:13.729365 21628 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.729867 21798 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.730648 21520 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.733036 21453 catalog_manager.cc:5526] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.736092 21453 catalog_manager.cc:5526] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.746925 21820 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.747584 21820 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.748045 21820 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.739799 21786 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.755764 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.759837 21746 tablet_bootstrap.cc:654] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.766497 21745 tablet_bootstrap.cc:654] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.769374 21787 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 08:58:13.755934 21786 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.767239 21701 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.780642 21744 tablet_bootstrap.cc:492] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.782222 21744 ts_tablet_manager.cc:1397] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.164s user 0.019s sys 0.013s
I20250114 08:58:13.776062 21798 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.788770 21744 raft_consensus.cc:357] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.789615 21786 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.789655 21744 raft_consensus.cc:383] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.790254 21744 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.791302 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.791115 21744 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.792214 21701 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:13.793450 21521 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:13.793745 21744 ts_tablet_manager.cc:1428] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250114 08:58:13.794314 21786 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:13.794664 21744 tablet_bootstrap.cc:492] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.794734 21786 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:13.795274 21786 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.790181 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:13.798390 21520 leader_election.cc:343] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.805932 21820 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.808727 21786 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.811712 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:13.813139 21520 leader_election.cc:343] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.815174 21786 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.816267 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.816938 21703 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.817906 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:13.818491 21628 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.824076 21703 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.825248 21521 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:13.826370 21786 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.826845 21786 raft_consensus.cc:695] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.827206 21798 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:13.827656 21786 consensus_queue.cc:237] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.834733 21628 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.835091 21746 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.835587 21746 ts_tablet_manager.cc:1397] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.200s user 0.018s sys 0.022s
I20250114 08:58:13.835893 21520 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:13.836731 21764 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:13.837203 21764 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:13.838158 21746 raft_consensus.cc:357] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.837997 21764 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.838918 21746 raft_consensus.cc:383] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.839248 21746 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.839865 21746 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.844955 21769 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.845381 21769 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.847096 21769 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.848148 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:13.849259 21596 leader_election.cc:343] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.849792 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.853178 21746 ts_tablet_manager.cc:1428] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20250114 08:58:13.852810 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:13.852980 21798 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 08:58:13.853892 21594 leader_election.cc:343] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.854360 21594 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.854792 21703 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.855875 21769 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.854091 21746 tablet_bootstrap.cc:492] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.858974 21450 catalog_manager.cc:5526] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.861317 21820 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.865223 21746 tablet_bootstrap.cc:654] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.865783 21744 tablet_bootstrap.cc:654] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.877709 21745 tablet_bootstrap.cc:492] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.878227 21745 ts_tablet_manager.cc:1397] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.243s user 0.010s sys 0.006s
I20250114 08:58:13.881004 21745 raft_consensus.cc:357] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.881826 21745 raft_consensus.cc:383] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.882133 21745 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.882731 21745 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.884274 21745 ts_tablet_manager.cc:1428] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 08:58:13.884956 21745 tablet_bootstrap.cc:492] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.889439 21745 tablet_bootstrap.cc:654] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.889864 21703 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:13.888993 21452 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:13.891377 21744 tablet_bootstrap.cc:492] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.891870 21744 ts_tablet_manager.cc:1397] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.000s
I20250114 08:58:13.894735 21744 raft_consensus.cc:357] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.895710 21769 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:13.896198 21744 raft_consensus.cc:383] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.896826 21744 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.900637 21744 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.903692 21744 ts_tablet_manager.cc:1428] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250114 08:58:13.906790 21746 tablet_bootstrap.cc:492] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.907265 21746 ts_tablet_manager.cc:1397] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.018s
I20250114 08:58:13.909760 21746 raft_consensus.cc:357] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.910538 21746 raft_consensus.cc:383] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.910806 21746 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.911826 21744 tablet_bootstrap.cc:492] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.911521 21746 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.919128 21746 ts_tablet_manager.cc:1428] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250114 08:58:13.920085 21746 tablet_bootstrap.cc:492] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.925611 21744 tablet_bootstrap.cc:654] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.926007 21746 tablet_bootstrap.cc:654] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.931020 21745 tablet_bootstrap.cc:492] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.931571 21745 ts_tablet_manager.cc:1397] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.005s
I20250114 08:58:13.933765 21787 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:13.934311 21745 raft_consensus.cc:357] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.935032 21745 raft_consensus.cc:383] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.935323 21745 raft_consensus.cc:738] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.935945 21745 consensus_queue.cc:260] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.938179 21745 ts_tablet_manager.cc:1428] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:58:13.940212 21745 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:13.943630 21744 tablet_bootstrap.cc:492] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.944103 21744 ts_tablet_manager.cc:1397] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.003s
I20250114 08:58:13.946609 21744 raft_consensus.cc:357] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.947307 21744 raft_consensus.cc:383] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.947537 21744 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.948084 21745 tablet_bootstrap.cc:654] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.948056 21744 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.949990 21744 ts_tablet_manager.cc:1428] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20250114 08:58:13.950855 21744 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.957857 21744 tablet_bootstrap.cc:654] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:13.959811 21746 tablet_bootstrap.cc:492] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:13.960291 21746 ts_tablet_manager.cc:1397] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.011s
I20250114 08:58:13.962786 21746 raft_consensus.cc:357] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.963522 21746 raft_consensus.cc:383] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.963794 21746 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.964869 21746 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.966787 21745 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:13.967347 21745 ts_tablet_manager.cc:1397] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.003s
I20250114 08:58:13.969183 21744 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:13.969539 21787 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.969647 21744 ts_tablet_manager.cc:1397] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20250114 08:58:13.970100 21787 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.970196 21745 raft_consensus.cc:357] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.971109 21745 raft_consensus.cc:383] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.971442 21745 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.972451 21744 raft_consensus.cc:357] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.972396 21745 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.973224 21744 raft_consensus.cc:383] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:13.973155 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.973711 21744 raft_consensus.cc:738] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:13.973922 21787 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.974575 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.974469 21744 consensus_queue.cc:260] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.975307 21703 raft_consensus.cc:2388] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:13.976788 21744 ts_tablet_manager.cc:1428] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:58:13.976831 21596 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.977790 21744 tablet_bootstrap.cc:492] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:13.978156 21787 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.980271 21751 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:13.980775 21751 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:13.982615 21746 ts_tablet_manager.cc:1428] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250114 08:58:13.982808 21751 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:13.984722 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:13.987959 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:13.988953 21703 raft_consensus.cc:2388] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:13.990146 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:13.990945 21769 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:13.994623 21746 tablet_bootstrap.cc:492] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:13.998381 21769 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:13.999943 21786 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.000507 21786 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.001238 21746 tablet_bootstrap.cc:654] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.002548 21744 tablet_bootstrap.cc:654] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.003494 21786 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.003445 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.004380 21628 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:14.005534 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.006284 21745 ts_tablet_manager.cc:1428] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250114 08:58:14.006358 21786 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.006479 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:14.006839 21786 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.007236 21786 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.007370 21745 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
W20250114 08:58:14.007819 21521 leader_election.cc:343] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:13.986719 21787 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:14.009951 21746 tablet_bootstrap.cc:492] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.010574 21746 ts_tablet_manager.cc:1397] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20250114 08:58:14.010720 21744 tablet_bootstrap.cc:492] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:14.011137 21744 ts_tablet_manager.cc:1397] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20250114 08:58:14.013239 21746 raft_consensus.cc:357] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.013763 21744 raft_consensus.cc:357] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.014019 21745 tablet_bootstrap.cc:654] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.014001 21746 raft_consensus.cc:383] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.014456 21744 raft_consensus.cc:383] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.014665 21746 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.015100 21786 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.014746 21744 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.015517 21746 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.018002 21746 ts_tablet_manager.cc:1428] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 08:58:14.019161 21746 tablet_bootstrap.cc:492] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.021371 21744 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.025167 21746 tablet_bootstrap.cc:654] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.027727 21744 ts_tablet_manager.cc:1428] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250114 08:58:14.028963 21744 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:14.036863 21744 tablet_bootstrap.cc:654] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.044291 21798 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.044890 21798 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.045183 21820 consensus_queue.cc:1035] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.046993 21798 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.046932 21820 consensus_queue.cc:1035] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:14.049439 21786 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.049556 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.050209 21630 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:14.050261 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.050997 21554 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:14.052052 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:14.054013 21521 leader_election.cc:343] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.054957 21764 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:14.055303 21745 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:14.055282 21669 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.055884 21745 ts_tablet_manager.cc:1397] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.049s user 0.021s sys 0.016s
I20250114 08:58:14.058591 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:14.059330 21627 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.058876 21745 raft_consensus.cc:357] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.059759 21745 raft_consensus.cc:383] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.060026 21745 raft_consensus.cc:738] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.060663 21771 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.061170 21771 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.061512 21771 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.060698 21745 consensus_queue.cc:260] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.068717 21746 tablet_bootstrap.cc:492] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.069609 21746 ts_tablet_manager.cc:1397] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.007s
I20250114 08:58:14.069664 21771 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.074478 21746 raft_consensus.cc:357] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.075119 21764 consensus_queue.cc:1035] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.076762 21798 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.077507 21745 ts_tablet_manager.cc:1428] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20250114 08:58:14.077533 21798 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.078572 21746 raft_consensus.cc:383] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.079223 21746 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.080374 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.081081 21554 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:14.082293 21668 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:14.083084 21745 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:14.083148 21752 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.083613 21752 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.083917 21752 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.084304 21764 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.084971 21764 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.088599 21764 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.088687 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.080534 21746 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.090246 21745 tablet_bootstrap.cc:654] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.092085 21746 ts_tablet_manager.cc:1428] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20250114 08:58:14.089449 21626 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:14.093775 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:14.100775 21702 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:14.101758 21746 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.102382 21520 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.097118 21798 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.096999 21764 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250114 08:58:14.103641 21786 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.104038 21786 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.104601 21786 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.099175 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.108876 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:14.109176 21669 leader_election.cc:343] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.109643 21554 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.113534 21746 tablet_bootstrap.cc:654] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.116789 21627 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:14.118062 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:14.118620 21752 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.119403 21764 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.119890 21764 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:14.120694 21752 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.121691 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:14.122514 21555 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.123278 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:14.124962 21669 leader_election.cc:343] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.121776 21764 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.131469 21555 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:14.131700 21746 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.132977 21668 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:14.133828 21746 ts_tablet_manager.cc:1397] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.000s
I20250114 08:58:14.134449 21752 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.136353 21771 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.136528 21786 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.135000 21752 raft_consensus.cc:695] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:14.138149 21554 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:14.138126 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:14.139683 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:14.139395 21752 consensus_queue.cc:237] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.140281 21629 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.139948 21627 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.143564 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:14.144332 21702 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.144732 21751 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.145334 21751 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.149044 21751 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.150221 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:14.151721 21704 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:14.150113 21668 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:14.152844 21596 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.153812 21751 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.154214 21751 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.154642 21751 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.162194 21771 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.162562 21820 consensus_queue.cc:1035] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.100762 21744 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:14.165756 21744 ts_tablet_manager.cc:1397] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.137s user 0.012s sys 0.009s
I20250114 08:58:14.162858 21771 raft_consensus.cc:695] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:14.171211 21771 consensus_queue.cc:237] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.142006 21786 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.147696 21746 raft_consensus.cc:357] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.172232 21751 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.182161 21629 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:14.186733 21746 raft_consensus.cc:383] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.212743 21746 raft_consensus.cc:738] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.213657 21746 consensus_queue.cc:260] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.190140 21744 raft_consensus.cc:357] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.217512 21744 raft_consensus.cc:383] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.217865 21744 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.218683 21744 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.195498 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:14.220887 21745 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:14.197135 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:14.195984 21751 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.221797 21744 ts_tablet_manager.cc:1428] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20250114 08:58:14.222333 21701 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.222824 21746 ts_tablet_manager.cc:1428] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.088s user 0.007s sys 0.001s
I20250114 08:58:14.223378 21744 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:14.223747 21746 tablet_bootstrap.cc:492] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.227909 21751 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.228564 21751 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.229585 21701 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:14.230876 21596 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.231678 21769 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.231889 21751 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.232203 21745 ts_tablet_manager.cc:1397] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.149s user 0.048s sys 0.054s
I20250114 08:58:14.232154 21769 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:14.233443 21769 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.234591 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:14.235304 21701 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:14.236398 21596 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.237238 21787 raft_consensus.cc:2798] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.237593 21787 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.237928 21787 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.238934 21745 raft_consensus.cc:357] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.239836 21745 raft_consensus.cc:383] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.240144 21745 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.241055 21745 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.243587 21745 ts_tablet_manager.cc:1428] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250114 08:58:14.244544 21745 tablet_bootstrap.cc:492] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:14.246325 21787 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.250407 21745 tablet_bootstrap.cc:654] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.250963 21798 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.251492 21798 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.255617 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.256742 21629 raft_consensus.cc:2388] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:14.256731 21744 tablet_bootstrap.cc:654] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.261744 21798 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:14.263391 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 79709.9us
I20250114 08:58:14.173480 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.271601 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:14.271956 21787 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.272332 21701 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.275538 21746 tablet_bootstrap.cc:654] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.280176 21701 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:14.281971 21596 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.283490 21787 raft_consensus.cc:2798] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.284143 21787 raft_consensus.cc:695] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:14.303320 21453 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:14.155982 21702 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
W20250114 08:58:14.333734 21594 leader_election.cc:343] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
W20250114 08:58:14.335915 21594 leader_election.cc:343] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.336313 21521 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:14.370941 21878 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.371618 21744 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:14.371726 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.372038 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:14.372205 21744 ts_tablet_manager.cc:1397] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.149s user 0.011s sys 0.004s
I20250114 08:58:14.372191 21878 raft_consensus.cc:695] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:14.373471 21878 consensus_queue.cc:237] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.375090 21744 raft_consensus.cc:357] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.375824 21744 raft_consensus.cc:383] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.376087 21744 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.376658 21554 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:14.376842 21744 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.378937 21744 ts_tablet_manager.cc:1428] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20250114 08:58:14.344558 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 124599us
I20250114 08:58:14.379868 21744 tablet_bootstrap.cc:492] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:14.369441 21787 consensus_queue.cc:237] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.386845 21744 tablet_bootstrap.cc:654] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.398229 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.407562 21668 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:14.409262 21752 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.409602 21871 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.411073 21871 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:14.413764 21744 tablet_bootstrap.cc:492] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:14.414243 21744 ts_tablet_manager.cc:1397] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.007s
I20250114 08:58:14.415957 21752 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:14.416800 21744 raft_consensus.cc:357] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.417466 21744 raft_consensus.cc:383] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.417682 21744 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.418094 21744 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.420120 21744 ts_tablet_manager.cc:1428] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20250114 08:58:14.421029 21744 tablet_bootstrap.cc:492] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:14.427556 21744 tablet_bootstrap.cc:654] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.427760 21627 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:14.445339 21870 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.445609 21453 catalog_manager.cc:5526] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:14.454844 21870 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:14.457676 21870 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 08:58:14.461180 21453 catalog_manager.cc:5526] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:14.455192 21746 tablet_bootstrap.cc:492] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.462989 21746 ts_tablet_manager.cc:1397] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.239s user 0.010s sys 0.003s
I20250114 08:58:14.465559 21746 raft_consensus.cc:357] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.468842 21746 raft_consensus.cc:383] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.469951 21746 raft_consensus.cc:738] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.471575 21746 consensus_queue.cc:260] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.474285 21873 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.479977 21745 tablet_bootstrap.cc:492] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:14.480424 21745 ts_tablet_manager.cc:1397] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.236s user 0.019s sys 0.020s
I20250114 08:58:14.482944 21745 raft_consensus.cc:357] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.483664 21745 raft_consensus.cc:383] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.483929 21745 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.486984 21746 ts_tablet_manager.cc:1428] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250114 08:58:14.488250 21746 tablet_bootstrap.cc:492] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.494872 21746 tablet_bootstrap.cc:654] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.503288 21451 catalog_manager.cc:5526] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:14.513314 21745 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.516638 21451 catalog_manager.cc:5526] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:14.535204 21745 ts_tablet_manager.cc:1428] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20250114 08:58:14.536592 21745 tablet_bootstrap.cc:492] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:14.544796 21453 catalog_manager.cc:5526] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:14.551928 21885 consensus_queue.cc:1035] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.555059 21885 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.555606 21885 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.558475 21885 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.558724 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.558852 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.559502 21554 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:14.560428 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:14.560976 21669 leader_election.cc:343] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.561264 21885 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.561640 21885 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.561954 21885 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.574960 21866 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.575492 21866 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.577382 21866 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.577697 21744 tablet_bootstrap.cc:492] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:14.578153 21744 ts_tablet_manager.cc:1397] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.157s user 0.047s sys 0.048s
I20250114 08:58:14.580055 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.580741 21746 tablet_bootstrap.cc:492] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.581236 21746 ts_tablet_manager.cc:1397] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.023s
W20250114 08:58:14.581410 21668 leader_election.cc:343] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.580742 21744 raft_consensus.cc:357] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.582667 21744 raft_consensus.cc:383] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.582937 21744 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.583539 21744 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.583691 21746 raft_consensus.cc:357] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.584417 21746 raft_consensus.cc:383] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.584755 21746 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.585498 21745 tablet_bootstrap.cc:654] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.585394 21746 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.586118 21744 ts_tablet_manager.cc:1428] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250114 08:58:14.586957 21744 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:14.587531 21746 ts_tablet_manager.cc:1428] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:58:14.584324 21885 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.589684 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.590040 21629 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:14.591050 21669 leader_election.cc:343] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.591640 21885 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:14.591521 21669 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:14.593138 21866 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:14.594134 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:14.595489 21669 leader_election.cc:343] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.602878 21885 consensus_queue.cc:1035] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.604350 21870 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:14.610303 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:14.611080 21554 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.613658 21744 tablet_bootstrap.cc:654] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.617691 21746 tablet_bootstrap.cc:492] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.619172 21745 tablet_bootstrap.cc:492] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:14.621654 21555 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.623906 21746 tablet_bootstrap.cc:654] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.627480 21554 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:14.628899 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:14.630496 21866 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.631152 21866 raft_consensus.cc:695] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:14.632275 21866 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.633534 21787 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.637890 21746 tablet_bootstrap.cc:492] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.638535 21746 ts_tablet_manager.cc:1397] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.001s
I20250114 08:58:14.641300 21746 raft_consensus.cc:357] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.642005 21746 raft_consensus.cc:383] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.642323 21746 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.642917 21746 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.644094 21769 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.647693 21746 ts_tablet_manager.cc:1428] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250114 08:58:14.649241 21746 tablet_bootstrap.cc:492] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.651966 21745 ts_tablet_manager.cc:1397] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.116s user 0.015s sys 0.018s
I20250114 08:58:14.668881 21745 raft_consensus.cc:357] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.669454 21746 tablet_bootstrap.cc:654] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.677511 21885 consensus_queue.cc:1035] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.687042 21745 raft_consensus.cc:383] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.687937 21745 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
W20250114 08:58:14.679245 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:14.689128 21745 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.692179 21745 ts_tablet_manager.cc:1428] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.039s user 0.004s sys 0.003s
I20250114 08:58:14.693428 21745 tablet_bootstrap.cc:492] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:14.695796 21452 catalog_manager.cc:5526] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:14.701282 21745 tablet_bootstrap.cc:654] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.719934 21745 tablet_bootstrap.cc:492] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:14.720531 21745 ts_tablet_manager.cc:1397] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.000s
I20250114 08:58:14.723799 21745 raft_consensus.cc:357] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.724794 21745 raft_consensus.cc:383] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.725154 21745 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.726811 21745 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.729900 21745 ts_tablet_manager.cc:1428] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 08:58:14.731158 21745 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:14.733855 21885 consensus_queue.cc:1035] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.739542 21745 tablet_bootstrap.cc:654] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.750882 21872 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.769357 21769 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.793454 21869 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250114 08:58:14.770149 21769 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.800284 21769 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.800268 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.801435 21554 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:14.801832 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:14.802506 21594 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.802508 21701 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:14.803468 21769 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.803844 21769 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.804191 21769 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.814056 21553 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.821452 21745 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:14.822117 21745 ts_tablet_manager.cc:1397] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.091s user 0.015s sys 0.016s
W20250114 08:58:14.828598 21899 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20250114 08:58:14.829378 21787 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.829365 21745 raft_consensus.cc:357] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.830211 21745 raft_consensus.cc:383] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.830497 21745 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.831105 21745 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.845554 21745 ts_tablet_manager.cc:1428] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20250114 08:58:14.846926 21745 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:14.831090 21769 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:14.861704 21895 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20250114 08:58:14.862350 21787 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.869971 21787 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.885466 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:14.886278 21552 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.888509 21769 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.890573 21787 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.891929 21746 tablet_bootstrap.cc:492] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.892168 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:14.892395 21746 ts_tablet_manager.cc:1397] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.243s user 0.041s sys 0.122s
W20250114 08:58:14.894531 21594 leader_election.cc:343] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.895133 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:14.896507 21746 raft_consensus.cc:357] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.897224 21746 raft_consensus.cc:383] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:14.897526 21746 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:14.898097 21746 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.898937 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:14.899560 21703 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.900283 21746 ts_tablet_manager.cc:1428] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
W20250114 08:58:14.897245 21596 leader_election.cc:343] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:14.901155 21596 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:14.901185 21746 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:14.902037 21751 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:14.905936 21764 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:14.910619 21552 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:14.911967 21594 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.913440 21787 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:14.913930 21787 raft_consensus.cc:695] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:14.914736 21787 consensus_queue.cc:237] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.920588 21744 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:14.921006 21744 ts_tablet_manager.cc:1397] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.334s user 0.064s sys 0.141s
I20250114 08:58:14.927976 21745 tablet_bootstrap.cc:654] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.933444 21769 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:14.959268 21703 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:14.963474 21870 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:14.968699 21746 tablet_bootstrap.cc:654] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:14.964737 21870 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:14.972450 21870 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:14.973513 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:14.974186 21629 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:14.977543 21520 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:14.985014 21870 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:14.986699 21870 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:14.987157 21870 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:14.990639 21746 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:14.991114 21746 ts_tablet_manager.cc:1397] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.010s
I20250114 08:58:14.985904 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:15.001108 21703 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:15.006903 21745 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.007397 21745 ts_tablet_manager.cc:1397] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.161s user 0.019s sys 0.016s
I20250114 08:58:15.007264 21746 raft_consensus.cc:357] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.007956 21746 raft_consensus.cc:383] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.007139 21870 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.009866 21870 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.011135 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:15.011776 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:15.009145 21744 raft_consensus.cc:357] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.011792 21701 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.020898 21629 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.014408 21745 raft_consensus.cc:357] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.021432 21745 raft_consensus.cc:383] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.021744 21745 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.022362 21745 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.023386 21744 raft_consensus.cc:383] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.008229 21746 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.025230 21746 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.027349 21745 ts_tablet_manager.cc:1428] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.020s user 0.010s sys 0.006s
I20250114 08:58:15.012073 21451 catalog_manager.cc:5526] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:15.027854 21746 ts_tablet_manager.cc:1428] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.036s user 0.001s sys 0.005s
I20250114 08:58:15.028311 21745 tablet_bootstrap.cc:492] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:15.028990 21787 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.029453 21787 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.031231 21746 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.023675 21744 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.033015 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:15.034039 21552 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:15.035135 21745 tablet_bootstrap.cc:654] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.035310 21787 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.037081 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:15.038679 21596 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:15.039947 21787 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.040918 21746 tablet_bootstrap.cc:654] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.041936 21629 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:15.043416 21520 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:15.043622 21701 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:15.044275 21870 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:15.045086 21870 raft_consensus.cc:695] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:15.039960 21744 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.045974 21870 consensus_queue.cc:237] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.048098 21787 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:15.048179 21744 ts_tablet_manager.cc:1428] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.127s user 0.007s sys 0.002s
I20250114 08:58:15.049952 21744 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.074440 21745 tablet_bootstrap.cc:492] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.074759 21744 tablet_bootstrap.cc:654] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.074971 21745 ts_tablet_manager.cc:1397] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.011s
I20250114 08:58:15.078824 21746 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.079365 21746 ts_tablet_manager.cc:1397] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.051s user 0.021s sys 0.004s
I20250114 08:58:15.081939 21746 raft_consensus.cc:357] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.082681 21746 raft_consensus.cc:383] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.083205 21744 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:15.083949 21745 raft_consensus.cc:357] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.084893 21744 ts_tablet_manager.cc:1397] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.000s
I20250114 08:58:15.085189 21745 raft_consensus.cc:383] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.085500 21745 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.086139 21745 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.088340 21745 ts_tablet_manager.cc:1428] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250114 08:58:15.088809 21744 raft_consensus.cc:357] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.089293 21745 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:15.089504 21744 raft_consensus.cc:383] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.089771 21744 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.090482 21744 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.091439 21452 catalog_manager.cc:5526] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:15.092772 21744 ts_tablet_manager.cc:1428] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:58:15.093907 21744 tablet_bootstrap.cc:492] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.095737 21745 tablet_bootstrap.cc:654] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.096520 21746 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.097458 21746 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.099589 21744 tablet_bootstrap.cc:654] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.099673 21746 ts_tablet_manager.cc:1428] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20250114 08:58:15.100589 21746 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.106364 21746 tablet_bootstrap.cc:654] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.117324 21745 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.117820 21745 ts_tablet_manager.cc:1397] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.001s
I20250114 08:58:15.118865 21744 tablet_bootstrap.cc:492] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:15.119328 21744 ts_tablet_manager.cc:1397] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.009s
I20250114 08:58:15.120100 21746 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.120786 21746 ts_tablet_manager.cc:1397] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.005s
I20250114 08:58:15.120540 21745 raft_consensus.cc:357] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.121381 21745 raft_consensus.cc:383] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.121650 21745 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.122009 21744 raft_consensus.cc:357] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.122262 21745 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.122697 21744 raft_consensus.cc:383] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.123066 21744 raft_consensus.cc:738] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.123836 21744 consensus_queue.cc:260] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.125038 21745 ts_tablet_manager.cc:1428] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250114 08:58:15.125757 21744 ts_tablet_manager.cc:1428] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 08:58:15.122864 21746 raft_consensus.cc:357] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.126085 21746 raft_consensus.cc:383] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.126593 21745 tablet_bootstrap.cc:492] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:15.126564 21746 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.127388 21746 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.129577 21746 ts_tablet_manager.cc:1428] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250114 08:58:15.130542 21746 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.132602 21745 tablet_bootstrap.cc:654] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.133242 21744 tablet_bootstrap.cc:492] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.139690 21798 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.140200 21798 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.142377 21798 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:15.143473 21746 tablet_bootstrap.cc:654] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.143474 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:15.144678 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:15.145314 21668 leader_election.cc:343] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
W20250114 08:58:15.146173 21669 leader_election.cc:343] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:15.146528 21669 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:15.147151 21798 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:15.150156 21744 tablet_bootstrap.cc:654] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.155586 21746 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.156140 21746 ts_tablet_manager.cc:1397] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.000s
I20250114 08:58:15.158640 21746 raft_consensus.cc:357] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.159406 21746 raft_consensus.cc:383] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.159716 21746 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.160408 21746 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.162853 21746 ts_tablet_manager.cc:1428] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 08:58:15.163760 21746 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.169521 21746 tablet_bootstrap.cc:654] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.181529 21744 tablet_bootstrap.cc:492] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:15.182013 21744 ts_tablet_manager.cc:1397] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.015s
I20250114 08:58:15.182559 21745 tablet_bootstrap.cc:492] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.183071 21745 ts_tablet_manager.cc:1397] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.057s user 0.033s sys 0.021s
I20250114 08:58:15.184188 21769 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.184669 21769 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.185547 21745 raft_consensus.cc:357] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.186229 21745 raft_consensus.cc:383] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.186489 21745 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.186657 21769 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.187189 21745 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.187942 21798 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:15.188189 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:15.189841 21596 leader_election.cc:343] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:15.191287 21744 raft_consensus.cc:357] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.191978 21744 raft_consensus.cc:383] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.192255 21744 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.192915 21744 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.194363 21745 ts_tablet_manager.cc:1428] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250114 08:58:15.195245 21745 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:15.196852 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:15.197521 21552 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:15.200183 21594 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:15.201678 21769 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:15.202046 21769 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:15.202391 21769 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.204267 21745 tablet_bootstrap.cc:654] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.207980 21798 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:15.209344 21746 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.209812 21746 ts_tablet_manager.cc:1397] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20250114 08:58:15.212409 21746 raft_consensus.cc:357] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.213126 21746 raft_consensus.cc:383] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.213387 21746 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.214531 21746 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.215695 21745 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.216153 21745 ts_tablet_manager.cc:1397] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.003s
I20250114 08:58:15.216732 21746 ts_tablet_manager.cc:1428] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250114 08:58:15.217635 21746 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.218935 21745 raft_consensus.cc:357] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.219624 21745 raft_consensus.cc:383] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.218807 21769 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.221678 21744 ts_tablet_manager.cc:1428] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20250114 08:58:15.221875 21769 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.222442 21744 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.222378 21745 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.223147 21745 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.224570 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:15.225461 21745 ts_tablet_manager.cc:1428] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20250114 08:58:15.226343 21745 tablet_bootstrap.cc:492] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
W20250114 08:58:15.226747 21596 leader_election.cc:343] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:15.227195 21744 tablet_bootstrap.cc:654] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.228116 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:15.228891 21555 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.237425 21745 tablet_bootstrap.cc:654] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.239674 21746 tablet_bootstrap.cc:654] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.243923 21555 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:15.244587 21745 tablet_bootstrap.cc:492] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.245005 21745 ts_tablet_manager.cc:1397] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.002s
I20250114 08:58:15.245380 21594 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:15.246754 21769 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:15.247210 21769 raft_consensus.cc:695] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:15.247421 21745 raft_consensus.cc:357] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.248139 21745 raft_consensus.cc:383] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.248447 21745 raft_consensus.cc:738] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.248004 21769 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.249047 21745 consensus_queue.cc:260] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.251322 21745 ts_tablet_manager.cc:1428] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:58:15.252157 21745 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:15.252164 21870 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.253858 21746 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.254271 21746 ts_tablet_manager.cc:1397] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.007s
I20250114 08:58:15.254107 21870 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.256045 21870 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.256731 21746 raft_consensus.cc:357] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.257421 21746 raft_consensus.cc:383] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.257670 21746 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.258260 21746 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.258726 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:15.260505 21521 leader_election.cc:343] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:15.261399 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:15.262022 21627 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:15.263257 21520 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:15.265731 21870 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:15.268028 21870 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:15.268637 21870 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.274472 21744 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:15.266372 21745 tablet_bootstrap.cc:654] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.260505 21746 ts_tablet_manager.cc:1428] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:58:15.274953 21744 ts_tablet_manager.cc:1397] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.009s
I20250114 08:58:15.275667 21746 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.277495 21744 raft_consensus.cc:357] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.278172 21744 raft_consensus.cc:383] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.278441 21744 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.278781 21870 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.279009 21744 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.281841 21744 ts_tablet_manager.cc:1428] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 08:58:15.282565 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:15.282852 21746 tablet_bootstrap.cc:654] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.283177 21701 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.286461 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:15.286873 21627 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.282895 21744 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.290506 21701 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:15.291581 21870 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.301255 21745 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.316562 21769 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.300117 21744 tablet_bootstrap.cc:654] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.293264 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:15.299049 21746 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.322630 21870 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:15.292410 21627 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:15.323184 21746 ts_tablet_manager.cc:1397] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.013s
I20250114 08:58:15.322392 21745 ts_tablet_manager.cc:1397] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.070s user 0.022s sys 0.012s
I20250114 08:58:15.322230 21769 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.325840 21746 raft_consensus.cc:357] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.326560 21746 raft_consensus.cc:383] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.326736 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:15.347470 21552 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:15.348767 21744 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:15.348985 21594 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:15.349340 21744 ts_tablet_manager.cc:1397] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.019s
I20250114 08:58:15.349797 21787 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:15.350127 21787 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:15.350394 21787 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.327419 21870 raft_consensus.cc:695] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:15.352596 21744 raft_consensus.cc:357] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.353302 21744 raft_consensus.cc:383] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.352403 21870 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.326980 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:15.357471 21701 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:15.326822 21746 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.353575 21744 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.379051 21746 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.330001 21769 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.380254 21744 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.383406 21745 raft_consensus.cc:357] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.383932 21745 raft_consensus.cc:383] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.384184 21745 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.385883 21746 ts_tablet_manager.cc:1428] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.062s user 0.005s sys 0.001s
I20250114 08:58:15.385795 21745 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.386752 21746 tablet_bootstrap.cc:492] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.393955 21751 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.394693 21751 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.401144 21745 ts_tablet_manager.cc:1428] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.077s user 0.007s sys 0.000s
I20250114 08:58:15.405956 21745 tablet_bootstrap.cc:492] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:15.410538 21744 ts_tablet_manager.cc:1428] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.061s user 0.008s sys 0.000s
I20250114 08:58:15.411980 21744 tablet_bootstrap.cc:492] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.431124 21751 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.434047 21787 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.442481 21745 tablet_bootstrap.cc:654] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.442941 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:15.436172 21787 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.443604 21701 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:15.442205 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:15.445741 21702 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.447417 21744 tablet_bootstrap.cc:654] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.449664 21746 tablet_bootstrap.cc:654] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.450194 21937 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:15.453675 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:15.454509 21551 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.455797 21596 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:15.456655 21787 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:15.457155 21937 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:15.457502 21937 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:15.457820 21937 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.461612 21746 tablet_bootstrap.cc:492] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:15.462101 21746 ts_tablet_manager.cc:1397] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.001s
I20250114 08:58:15.464730 21746 raft_consensus.cc:357] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.456091 21744 tablet_bootstrap.cc:492] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:15.465481 21746 raft_consensus.cc:383] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.466292 21746 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.466676 21744 ts_tablet_manager.cc:1397] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.004s
I20250114 08:58:15.470042 21746 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.468297 21702 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:15.478340 21596 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:15.478828 21746 ts_tablet_manager.cc:1428] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250114 08:58:15.479436 21769 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:15.479951 21769 raft_consensus.cc:695] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:15.481034 21769 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.482381 21937 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.483134 21744 raft_consensus.cc:357] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.485139 21771 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.485445 21744 raft_consensus.cc:383] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.479863 21746 tablet_bootstrap.cc:492] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:15.485638 21771 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.485869 21744 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.486478 21938 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.486984 21938 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.487748 21771 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:15.489104 21938 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.489317 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:15.489864 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:15.489811 21744 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.491755 21771 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.492208 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:15.492274 21771 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.490639 21703 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.496892 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:15.497606 21627 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:15.498976 21669 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:15.500030 21885 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:15.489969 21629 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:15.500437 21885 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:15.500876 21885 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:15.501456 21596 leader_election.cc:343] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:15.502056 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:15.503064 21866 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:15.503479 21866 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:15.503822 21866 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.506294 21551 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:15.507375 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:15.507787 21870 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.508072 21551 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:15.508323 21870 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.509438 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:15.510493 21870 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.510995 21746 tablet_bootstrap.cc:654] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:15.511657 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:15.512804 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:15.515053 21594 leader_election.cc:343] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:15.515547 21594 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:15.516518 21787 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:15.524667 21771 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:15.525408 21702 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:15.490586 21937 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.545115 21885 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.547760 21745 tablet_bootstrap.cc:492] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:15.503834 21703 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:15.548199 21885 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:15.548329 21745 ts_tablet_manager.cc:1397] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.146s user 0.010s sys 0.001s
I20250114 08:58:15.547637 21866 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.550314 21866 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:15.551182 21744 ts_tablet_manager.cc:1428] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.082s user 0.006s sys 0.002s
I20250114 08:58:15.552089 21744 tablet_bootstrap.cc:492] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:15.553872 21745 raft_consensus.cc:357] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.555083 21745 raft_consensus.cc:383] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:15.555361 21745 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:15.555881 21745 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.560618 21551 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
W20250114 08:58:15.747977 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:15.790339 21937 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:15.790899 21937 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:15.792925 21937 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:15.793530 21744 tablet_bootstrap.cc:654] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:16.177250 21453 catalog_manager.cc:5526] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:16.192376 21594 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.197602 21769 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:16.198184 21769 raft_consensus.cc:695] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:16.199112 21769 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.252272 21866 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.253659 21866 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:16.278885 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 86590.3us
I20250114 08:58:16.309484 21866 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:16.315971 21746 tablet_bootstrap.cc:492] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:16.316510 21746 ts_tablet_manager.cc:1397] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.837s user 0.011s sys 0.022s
I20250114 08:58:16.319128 21746 raft_consensus.cc:357] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.320111 21746 raft_consensus.cc:383] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:16.320446 21746 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:16.324529 21746 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.333616 21745 ts_tablet_manager.cc:1428] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.785s user 0.014s sys 0.035s
W20250114 08:58:16.334108 21745 ts_tablet_manager.cc:1460] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Tablet startup took 785ms
I20250114 08:58:16.339632 21746 ts_tablet_manager.cc:1428] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20250114 08:58:16.341188 21746 tablet_bootstrap.cc:492] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:16.345737 21868 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.346371 21868 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:16.337635 21745 ts_tablet_manager.cc:1462] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Trace:
0114 08:58:12.140647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:12.140784 (+ 137us) service_pool.cc:225] Handling call
0114 08:58:12.146292 (+ 5508us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0114 08:58:12.146385 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0114 08:58:12.190987 (+ 44602us) tablet_metadata.cc:641] Metadata flushed
0114 08:58:12.292963 (+101976us) tablet_replica.cc:206] Creating consensus instance
0114 08:58:12.293482 (+ 519us) inbound_call.cc:173] Queueing success response
0114 08:58:15.402188 (+3108706us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0114 08:58:15.548881 (+146693us) ts_tablet_manager.cc:1429] Starting tablet replica
0114 08:58:15.549444 (+ 563us) tablet_replica.cc:260] Starting instrumentation
0114 08:58:15.551920 (+ 2476us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":765960,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":5225,"tablet-open.queue_time_us":3108808}
I20250114 08:58:16.361936 21745 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:16.350080 21868 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.370702 21744 tablet_bootstrap.cc:492] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:16.371374 21744 ts_tablet_manager.cc:1397] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.819s user 0.014s sys 0.003s
I20250114 08:58:16.407620 21744 raft_consensus.cc:357] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.408641 21744 raft_consensus.cc:383] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:16.408990 21744 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:16.409839 21744 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.419071 21746 tablet_bootstrap.cc:654] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:16.426245 21744 ts_tablet_manager.cc:1428] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.055s user 0.008s sys 0.001s
I20250114 08:58:16.431746 21745 tablet_bootstrap.cc:654] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:16.436280 21744 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:16.420907 21769 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.422623 21937 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.420751 21950 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.423712 21787 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.435365 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:16.444764 21937 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.454588 21451 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:16.449349 21629 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.445955 21787 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.558740 21937 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.559268 21937 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.446614 21950 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.559765 21937 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.561106 21950 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.442431 21769 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.448432 21955 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.562467 21875 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.566417 21955 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.514317 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:16.563477 21787 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.565680 21950 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.593673 21948 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.592299 21751 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:16.595753 21627 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.600239 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.600878 21866 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.602795 21764 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.604897 21937 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.604396 21870 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:16.594962 21875 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.625985 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.604274 21948 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.626721 21626 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:16.599371 21950 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:16.605567 21751 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.642748 21875 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.643604 21769 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.639125 21960 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.645104 21948 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:16.618278 21866 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.631358 21961 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.625077 21870 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.653266 21961 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.621230 21764 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.659384 21744 tablet_bootstrap.cc:654] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:16.651854 21868 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:16.644361 21960 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:16.663187 21576 debug-util.cc:398] Leaking SignalData structure 0x7b08000dfb20 after lost signal to thread 20385
I20250114 08:58:16.667104 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:16.667639 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800114c40 after lost signal to thread 21487
I20250114 08:58:16.667912 21960 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:16.668807 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080026a0e0 after lost signal to thread 21579
W20250114 08:58:16.669134 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080026a100 after lost signal to thread 21580
I20250114 08:58:16.669471 21870 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:16.670078 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080026a920 after lost signal to thread 21654
W20250114 08:58:16.670385 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080026a940 after lost signal to thread 21655
I20250114 08:58:16.671062 21771 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:58:16.671311 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800173600 after lost signal to thread 21728
I20250114 08:58:16.660326 21961 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.671602 21771 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.672317 21955 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.672590 21866 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:16.671626 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800134960 after lost signal to thread 21729
I20250114 08:58:16.673578 21771 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:16.673796 21764 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:16.673969 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800219fe0 after lost signal to thread 21915
I20250114 08:58:16.659934 21751 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.665036 21868 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.677310 21745 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:16.663833 21948 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.663231 21965 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:16.681391 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.681576 21950 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:16.680008 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.682233 21745 ts_tablet_manager.cc:1397] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.321s user 0.015s sys 0.003s
I20250114 08:58:16.683204 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.681423 21948 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.683871 21550 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:16.684114 21629 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:16.690555 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.691779 21868 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:16.679529 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800219f40 after lost signal to thread 21920
I20250114 08:58:16.694675 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.695796 21950 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.697500 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.696300 21950 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.699762 21950 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.704511 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.702564 21948 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:16.705264 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.707108 21669 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.708180 21866 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:16.708351 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.708596 21866 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:16.708983 21866 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.702488 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.710148 21624 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:16.711189 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.681635 21965 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.715564 21965 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.693054 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.716645 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.696691 21751 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:16.717404 21751 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.705319 21703 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:16.717860 21866 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.718732 21950 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:16.719874 21866 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:16.718246 21596 consensus_peers.cc:487] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed -> Peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Couldn't send request to peer cec0ef3bbb654988bf2123d8eef7a0a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250114 08:58:16.720659 21596 leader_election.cc:343] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:16.721284 21669 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.722316 21866 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:16.723006 21866 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:16.723649 21745 raft_consensus.cc:357] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.724313 21745 raft_consensus.cc:383] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:16.723098 21751 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.724586 21745 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:16.725651 21745 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.728986 21521 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:16.730532 21746 tablet_bootstrap.cc:492] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:16.732059 21627 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:16.722635 21669 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.733001 21866 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.733644 21771 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.738209 21771 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.734951 21745 ts_tablet_manager.cc:1428] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.052s user 0.003s sys 0.002s
I20250114 08:58:16.735569 21746 ts_tablet_manager.cc:1397] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.395s user 0.019s sys 0.004s
I20250114 08:58:16.741142 21745 tablet_bootstrap.cc:492] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:16.743410 21744 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:16.744037 21744 ts_tablet_manager.cc:1397] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.309s user 0.005s sys 0.009s
I20250114 08:58:16.747092 21948 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:16.740094 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.747493 21948 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.745450 21965 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:16.749253 21965 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:16.749727 21965 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.751058 21746 raft_consensus.cc:357] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.751816 21746 raft_consensus.cc:383] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:16.752118 21746 raft_consensus.cc:738] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:16.747898 21948 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:16.757472 21745 tablet_bootstrap.cc:654] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:16.758335 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.759450 21746 consensus_queue.cc:260] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.758986 21624 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:16.761385 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.762017 21552 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:16.763034 21771 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:16.760696 21744 raft_consensus.cc:357] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.767933 21744 raft_consensus.cc:383] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:16.768222 21744 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:16.770833 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.771520 21553 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:16.772979 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.773761 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.774205 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.774533 21703 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:16.776294 21744 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.776774 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:16.777320 21553 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.777588 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:16.778174 21550 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.778332 21744 ts_tablet_manager.cc:1428] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250114 08:58:16.779300 21744 tablet_bootstrap.cc:492] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:16.780589 21948 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.781615 21746 ts_tablet_manager.cc:1428] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20250114 08:58:16.782532 21746 tablet_bootstrap.cc:492] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
W20250114 08:58:16.786594 21521 leader_election.cc:343] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:16.787524 21868 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:16.789299 21964 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:16.791294 21948 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:16.784171 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.794065 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.802933 21550 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:16.803834 21453 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 51) took 1468 ms. Trace:
I20250114 08:58:16.805488 21452 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55888 (request call id 52) took 1278 ms. Trace:
I20250114 08:58:16.783877 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.804862 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:16.805274 21521 leader_election.cc:343] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:16.812321 21948 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:16.806123 21452 rpcz_store.cc:276] 0114 08:58:15.527032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:15.537064 (+ 10032us) service_pool.cc:225] Handling call
0114 08:58:16.805407 (+1268343us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":252398,"reported_tablets":1}
I20250114 08:58:16.812948 21948 raft_consensus.cc:695] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:16.814185 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.813958 21948 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.806010 21453 rpcz_store.cc:276] 0114 08:58:15.335855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:15.336014 (+ 159us) service_pool.cc:225] Handling call
0114 08:58:16.803746 (+1467732us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:16.191087 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:16.191606 (+ 519us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:16.191639 (+ 33us) write_op.cc:435] Acquired schema lock
0114 08:58:16.191657 (+ 18us) tablet.cc:659] Decoding operations
0114 08:58:16.191978 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:16.192033 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:16.192060 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0114 08:58:16.192242 (+ 182us) tablet.cc:698] Row locks acquired
0114 08:58:16.192260 (+ 18us) write_op.cc:260] PREPARE: finished
0114 08:58:16.192398 (+ 138us) write_op.cc:270] Start()
0114 08:58:16.192544 (+ 146us) write_op.cc:276] Timestamp: P: 1736845096192359 usec, L: 0
0114 08:58:16.192574 (+ 30us) op_driver.cc:348] REPLICATION: starting
0114 08:58:16.193270 (+ 696us) log.cc:844] Serialized 543 byte log entry
0114 08:58:16.596304 (+403034us) op_driver.cc:464] REPLICATION: finished
0114 08:58:16.713968 (+117664us) write_op.cc:301] APPLY: starting
0114 08:58:16.723128 (+ 9160us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:16.723230 (+ 102us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:16.723351 (+ 121us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:16.727024 (+ 3673us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:16.740331 (+ 13307us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 08:58:16.745320 (+ 4989us) write_op.cc:312] APPLY: finished
0114 08:58:16.783113 (+ 37793us) log.cc:844] Serialized 22 byte log entry
0114 08:58:16.785840 (+ 2727us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:16.788669 (+ 2829us) write_op.cc:454] Released schema lock
0114 08:58:16.789975 (+ 1306us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:16.801420 (+ 11445us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:16.801452 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3881,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":96352,"apply.run_cpu_time_us":2040,"apply.run_wall_time_us":97286,"num_ops":1,"prepare.queue_time_us":10558,"prepare.run_cpu_time_us":27104,"prepare.run_wall_time_us":101810,"raft.queue_time_us":289083,"raft.run_cpu_time_us":47807,"raft.run_wall_time_us":154129,"replication_time_us":377743,"thread_start_us":252449,"threads_started":3,"wal-append.queue_time_us":117676}]]}
I20250114 08:58:16.820084 21744 tablet_bootstrap.cc:654] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:16.805116 21627 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:16.816694 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.824976 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.825754 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:16.826362 21630 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.821771 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.827873 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:16.827975 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.830675 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.826857 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.822230 21965 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.822858 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.833287 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.832639 21751 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:16.833854 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.833940 21751 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:16.833956 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.834509 21751 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:16.834666 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.835480 21701 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:16.836256 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.838697 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.838250 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.839995 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:16.839438 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.841320 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.841562 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:16.842633 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:16.842981 21950 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:16.843520 21950 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:16.843940 21950 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:16.847813 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:16.857477 21965 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:16.836603 21521 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:16.860636 21630 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:16.864511 21965 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:16.862483 21938 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250114 08:58:16.872948 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:16.873291 21594 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:16.873584 21938 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:16.875947 21962 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:16.876359 21962 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:16.876708 21962 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.881551 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.868045 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.021136 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:16.886794 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.992235 21953 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:16.967691 21973 log.cc:926] Time spent T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20250114 08:58:17.043692 21699 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.044369 21953 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.887224 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:17.000636 21668 leader_election.cc:343] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
W20250114 08:58:17.047317 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 46733.5us
I20250114 08:58:16.995896 21669 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:17.026274 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:16.994398 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:17.048960 21947 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.049227 21630 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.049754 21947 raft_consensus.cc:695] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:58:16.954926 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:17.028147 21703 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:16.992611 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:17.050971 21947 consensus_queue.cc:237] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.954936 21745 tablet_bootstrap.cc:492] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.052600 21745 ts_tablet_manager.cc:1397] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.312s user 0.013s sys 0.013s
W20250114 08:58:17.053609 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 57837.7us
I20250114 08:58:16.987720 21948 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:16.992870 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.012899 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.059130 21624 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:16.874872 21965 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.065191 21965 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:16.887138 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:16.869155 21553 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:16.984647 21751 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:16.903496 21950 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.022275 21962 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.071301 21950 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.072144 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.072554 21965 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.073673 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.054693 21948 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.074791 21965 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.073151 21751 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.074668 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.075637 21702 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.076779 21699 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:17.081080 21702 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:17.081825 21751 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.082051 21594 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.086011 21953 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:17.039860 21744 tablet_bootstrap.cc:492] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:17.087658 21744 ts_tablet_manager.cc:1397] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.309s user 0.017s sys 0.002s
I20250114 08:58:17.088476 21962 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:17.088677 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 52340.6us
I20250114 08:58:17.089838 21630 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:17.090487 21521 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:17.091511 21868 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:17.092619 21868 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.093050 21868 raft_consensus.cc:695] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:17.093415 21959 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:17.093854 21959 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:17.093928 21868 consensus_queue.cc:237] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.094831 21959 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.096421 21959 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:17.097960 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 49552us
I20250114 08:58:17.042658 21746 tablet_bootstrap.cc:654] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.101665 21751 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.100807 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.103298 21701 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.105304 21948 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:17.105626 21744 raft_consensus.cc:357] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.107784 21744 raft_consensus.cc:383] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.108078 21744 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.108731 21744 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.115370 21745 raft_consensus.cc:357] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.116982 21596 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:17.119798 21668 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.120977 21866 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.122018 21668 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.115929 21745 raft_consensus.cc:383] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.123557 21745 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.124230 21745 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.127440 21744 ts_tablet_manager.cc:1428] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.039s user 0.005s sys 0.002s
I20250114 08:58:17.128319 21668 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.129061 21744 tablet_bootstrap.cc:492] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:17.129563 21668 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.130337 21701 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:17.131692 21521 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:17.124709 21866 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.133117 21771 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.137686 21964 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.138094 21868 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.138185 21964 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:17.138620 21868 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.139227 21964 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.140003 21745 ts_tablet_manager.cc:1428] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.087s user 0.005s sys 0.001s
I20250114 08:58:17.137734 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.133456 21453 catalog_manager.cc:5526] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:17.142445 21963 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:17.126590 21962 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.142275 21453 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:17.141016 21745 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:17.142803 21963 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.142138 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:17.152648 21958 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250114 08:58:17.134778 21947 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.154327 21963 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.140667 21868 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.154342 21962 raft_consensus.cc:695] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:17.163861 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:17.165120 21520 leader_election.cc:343] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.142997 21594 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:17.154989 21668 leader_election.cc:343] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.166206 21764 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.168744 21594 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.144272 21955 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250114 08:58:17.169631 21960 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.170214 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.180680 21950 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.186698 21764 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.178387 21947 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.199926 21962 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.200919 21594 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.205034 21950 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.206393 21520 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:17.208283 21520 leader_election.cc:343] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.208542 21955 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.209144 21744 tablet_bootstrap.cc:654] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.209471 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.198969 21947 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.218017 21745 tablet_bootstrap.cc:654] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.213990 21947 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:17.213145 21520 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.219178 21990 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.221149 21955 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.221724 21746 tablet_bootstrap.cc:492] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:17.221836 21594 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.222277 21746 ts_tablet_manager.cc:1397] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.440s user 0.018s sys 0.007s
I20250114 08:58:17.225102 21668 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.226061 21771 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.226677 21594 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.226872 21746 raft_consensus.cc:357] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.227603 21746 raft_consensus.cc:383] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.227885 21746 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.228031 21950 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.228542 21746 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.223744 21962 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.231662 21746 ts_tablet_manager.cc:1428] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 08:58:17.232424 21746 tablet_bootstrap.cc:492] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:17.236979 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:17.238421 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.239300 21629 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:17.227896 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.246603 21962 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.247880 21629 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 08:58:17.262734 21594 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.265587 21744 tablet_bootstrap.cc:492] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:17.267817 21746 tablet_bootstrap.cc:654] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.267728 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.268755 21744 ts_tablet_manager.cc:1397] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.137s user 0.020s sys 0.022s
I20250114 08:58:17.270327 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.272720 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.273252 21551 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:17.268831 21451 catalog_manager.cc:5526] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:17.277166 21962 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:17.275805 21520 leader_election.cc:343] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.275674 21947 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:17.276288 21668 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:17.282191 21962 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.283298 21947 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:17.283723 21947 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.284096 21947 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.283952 21744 raft_consensus.cc:357] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.285732 21744 raft_consensus.cc:383] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.286001 21744 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.286159 21451 catalog_manager.cc:5526] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:17.286618 21744 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.285107 21963 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.290958 21744 ts_tablet_manager.cc:1428] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.022s user 0.003s sys 0.005s
I20250114 08:58:17.291033 21553 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:17.289978 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.292121 21962 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.293264 21744 tablet_bootstrap.cc:492] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:17.294936 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.298228 21947 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.305002 21451 catalog_manager.cc:5526] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:17.304360 21745 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.307839 21745 ts_tablet_manager.cc:1397] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.167s user 0.008s sys 0.016s
I20250114 08:58:17.309450 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.310525 21764 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.311233 21964 consensus_queue.cc:1046] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.312510 21868 raft_consensus.cc:3049] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:17.312903 21868 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:17.313728 21868 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.315050 21868 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:17.315290 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.317723 21553 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:17.321530 21745 raft_consensus.cc:357] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.322275 21745 raft_consensus.cc:383] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.322566 21745 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.323415 21745 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.325816 21745 ts_tablet_manager.cc:1428] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.018s user 0.002s sys 0.003s
I20250114 08:58:17.329022 21453 catalog_manager.cc:5526] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:17.331446 21764 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.332196 21745 tablet_bootstrap.cc:492] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:17.331372 21453 catalog_manager.cc:5526] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:17.339358 21963 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.308750 21947 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:17.336757 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.346068 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:17.362478 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.363396 21700 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:17.364308 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.365448 21553 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.366431 21746 tablet_bootstrap.cc:492] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:17.366955 21746 ts_tablet_manager.cc:1397] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.005s
I20250114 08:58:17.369665 21746 raft_consensus.cc:357] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.370406 21746 raft_consensus.cc:383] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.370682 21746 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.372998 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.373410 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.371878 21746 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.374365 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.375000 21628 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:17.379570 21963 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.380126 21963 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.380656 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.381783 21744 tablet_bootstrap.cc:654] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.382172 21963 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.383842 21964 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.379609 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:17.387482 21629 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.388610 21746 ts_tablet_manager.cc:1428] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250114 08:58:17.389667 21746 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:17.390229 21553 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:17.390357 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:17.393352 21521 leader_election.cc:343] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.398640 21624 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:17.402343 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.409454 21629 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:17.411160 21668 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:17.413890 21866 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.414467 21745 tablet_bootstrap.cc:654] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.414410 21866 raft_consensus.cc:695] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:17.415484 21866 consensus_queue.cc:237] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.420887 21948 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:17.421723 21964 consensus_queue.cc:1046] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:17.423245 21963 raft_consensus.cc:3049] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:17.423614 21963 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:17.424408 21963 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.426115 21963 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:17.426505 21520 leader_election.cc:343] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.426906 21520 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.427052 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.427537 21964 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.428694 21947 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.429883 21669 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.431485 21947 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.432662 21948 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.433178 21948 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.435801 21948 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:17.436329 21948 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250114 08:58:17.437811 21594 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.441038 21453 catalog_manager.cc:5526] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:17.446108 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.443490 21960 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250114 08:58:17.449551 21668 leader_election.cc:343] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.451138 21594 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.454859 21769 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:17.459972 21744 tablet_bootstrap.cc:492] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:17.460702 21744 ts_tablet_manager.cc:1397] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.168s user 0.015s sys 0.008s
I20250114 08:58:17.463172 21451 catalog_manager.cc:5526] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:17.467103 21746 tablet_bootstrap.cc:654] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.481590 21744 raft_consensus.cc:357] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.482407 21744 raft_consensus.cc:383] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.482702 21744 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.483891 21744 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.486763 21744 ts_tablet_manager.cc:1428] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20250114 08:58:17.487748 21744 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:17.492810 21948 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:17.493268 21745 tablet_bootstrap.cc:492] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.494565 21745 ts_tablet_manager.cc:1397] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.168s user 0.013s sys 0.008s
I20250114 08:58:17.500860 21745 raft_consensus.cc:357] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.501770 21745 raft_consensus.cc:383] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.502063 21745 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.502835 21745 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.506218 21745 ts_tablet_manager.cc:1428] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250114 08:58:17.507058 21745 tablet_bootstrap.cc:492] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:17.516178 21746 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:17.516742 21746 ts_tablet_manager.cc:1397] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.127s user 0.014s sys 0.002s
I20250114 08:58:17.516427 21963 consensus_queue.cc:1035] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.520030 21746 raft_consensus.cc:357] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.520853 21746 raft_consensus.cc:383] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.521153 21746 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.522444 21746 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.520298 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.524811 21746 ts_tablet_manager.cc:1428] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250114 08:58:17.529501 21629 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:17.551784 21746 tablet_bootstrap.cc:492] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:17.541509 21744 tablet_bootstrap.cc:654] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.556777 21948 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.556969 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:17.558811 21948 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:17.560765 21948 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.566335 21948 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.568588 21745 tablet_bootstrap.cc:654] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.577615 21769 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.578104 21769 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.581128 21769 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.586107 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.586640 21551 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:17.591245 21594 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:17.592123 21769 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:17.592512 21769 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.592839 21769 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.592715 21866 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:17.595614 21948 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.598942 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:17.604588 21948 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:17.605297 21596 leader_election.cc:343] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.605921 21746 tablet_bootstrap.cc:654] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.610129 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.611027 21990 consensus_queue.cc:1035] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.611651 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:17.612342 21624 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:17.616519 21668 leader_election.cc:343] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.624909 21764 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:17.626662 21624 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:17.638832 21745 tablet_bootstrap.cc:492] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.639605 21745 ts_tablet_manager.cc:1397] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.133s user 0.013s sys 0.018s
I20250114 08:58:17.642066 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:17.651823 21769 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.653955 21769 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.654979 21744 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:17.655479 21744 ts_tablet_manager.cc:1397] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.168s user 0.009s sys 0.009s
I20250114 08:58:17.656144 21948 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.658640 21948 raft_consensus.cc:695] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:17.658903 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.659561 21554 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.659512 21948 consensus_queue.cc:237] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.662249 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:17.663573 21596 leader_election.cc:343] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.666684 21744 raft_consensus.cc:357] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.667454 21744 raft_consensus.cc:383] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.667734 21744 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.668757 21744 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.674306 21554 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:17.676393 21594 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.679672 21769 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.680136 21769 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:17.680881 21769 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.700747 21745 raft_consensus.cc:357] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.712642 21745 raft_consensus.cc:383] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.713099 21745 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.702243 21744 ts_tablet_manager.cc:1428] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250114 08:58:17.705657 21771 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.714893 21744 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:17.715901 21771 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.718657 21771 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:17.723268 21745 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.728873 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:17.731369 22019 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20250114 08:58:17.721616 21453 catalog_manager.cc:5526] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
W20250114 08:58:17.733783 21668 leader_election.cc:343] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.731865 21960 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250114 08:58:17.741492 21745 ts_tablet_manager.cc:1428] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.101s user 0.008s sys 0.000s
I20250114 08:58:17.732904 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:17.743499 21745 tablet_bootstrap.cc:492] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
W20250114 08:58:17.748363 21669 leader_election.cc:343] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.748821 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:17.749573 21771 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:17.744313 21746 tablet_bootstrap.cc:492] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:17.750311 21746 ts_tablet_manager.cc:1397] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.199s user 0.024s sys 0.039s
W20250114 08:58:17.745285 21520 consensus_peers.cc:487] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Couldn't send request to peer d7732324fb5f48d3b16a29d4bb2584ed. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:17.744705 21452 catalog_manager.cc:5526] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:17.763211 21744 tablet_bootstrap.cc:654] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.766006 21746 raft_consensus.cc:357] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.770294 21746 raft_consensus.cc:383] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.770622 21746 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.771402 21746 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.773943 21746 ts_tablet_manager.cc:1428] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250114 08:58:17.774911 21746 tablet_bootstrap.cc:492] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:17.787067 21745 tablet_bootstrap.cc:654] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.789806 21962 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.798278 21746 tablet_bootstrap.cc:654] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.804965 21745 tablet_bootstrap.cc:492] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.805465 21745 ts_tablet_manager.cc:1397] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.006s
I20250114 08:58:17.807865 21769 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.808389 21769 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.809887 21745 raft_consensus.cc:357] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.810362 21769 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.810642 21745 raft_consensus.cc:383] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.810958 21745 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.811986 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:17.813602 21596 leader_election.cc:343] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.810832 21746 tablet_bootstrap.cc:492] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:17.815819 21746 ts_tablet_manager.cc:1397] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.001s
I20250114 08:58:17.811589 21745 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.818442 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.828233 21551 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:17.829604 21594 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.830407 21769 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:17.830865 21769 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.831226 21769 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.852164 21745 ts_tablet_manager.cc:1428] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.046s user 0.002s sys 0.015s
I20250114 08:58:17.853331 21745 tablet_bootstrap.cc:492] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:17.857722 21746 raft_consensus.cc:357] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.858458 21746 raft_consensus.cc:383] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.858718 21746 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.859408 21746 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.860191 21745 tablet_bootstrap.cc:654] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.867803 21746 ts_tablet_manager.cc:1428] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.052s user 0.010s sys 0.008s
I20250114 08:58:17.869966 21746 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:17.880288 21746 tablet_bootstrap.cc:654] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.886482 21769 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.890444 21699 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.892745 21745 tablet_bootstrap.cc:492] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.906301 21745 ts_tablet_manager.cc:1397] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.007s
I20250114 08:58:17.909047 21745 raft_consensus.cc:357] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.909838 21745 raft_consensus.cc:383] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.910158 21745 raft_consensus.cc:738] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.910797 21745 consensus_queue.cc:260] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.895403 21744 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:17.912091 21744 ts_tablet_manager.cc:1397] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.197s user 0.009s sys 0.039s
I20250114 08:58:17.913347 21745 ts_tablet_manager.cc:1428] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 08:58:17.913973 21769 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.914640 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.915030 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.917955 21551 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.918473 21745 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
W20250114 08:58:17.918895 21596 leader_election.cc:343] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:17.924572 21551 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:17.925806 21594 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:17.926609 21962 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:17.927124 21745 tablet_bootstrap.cc:654] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.927088 21962 raft_consensus.cc:695] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:17.917088 21744 raft_consensus.cc:357] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.928270 21962 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.936025 21746 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:17.936620 21746 ts_tablet_manager.cc:1397] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.016s
I20250114 08:58:17.928241 21744 raft_consensus.cc:383] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.939671 21744 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.939339 21746 raft_consensus.cc:357] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.940995 21746 raft_consensus.cc:383] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.941320 21746 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.941802 21744 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.947853 21744 ts_tablet_manager.cc:1428] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20250114 08:58:17.950699 21744 tablet_bootstrap.cc:492] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:17.952373 21746 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.957194 21746 ts_tablet_manager.cc:1428] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250114 08:58:17.958191 21746 tablet_bootstrap.cc:492] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:17.963347 21769 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:17.964210 21744 tablet_bootstrap.cc:654] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.964006 21769 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.966408 21769 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:17.972136 21746 tablet_bootstrap.cc:654] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:17.975153 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:17.975373 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:17.977391 21745 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:17.976136 21551 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:17.977691 21699 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:17.977912 21745 ts_tablet_manager.cc:1397] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.016s
I20250114 08:58:17.978909 21594 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:17.979923 21769 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:17.980226 21769 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:17.980535 21769 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.987555 21769 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.990859 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:17.991531 21699 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:17.993952 21745 raft_consensus.cc:357] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.994704 21745 raft_consensus.cc:383] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:17.995075 21745 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:17.996069 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:17.996778 21551 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.009806 21769 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.011617 22008 consensus_queue.cc:1035] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:17.989292 21451 catalog_manager.cc:5526] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:18.012367 21699 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:18.017467 21596 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.018666 21948 consensus_queue.cc:1035] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:18.017421 21551 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:18.027122 21962 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:18.027668 21962 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:18.028522 21962 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:17.995904 21745 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.032024 21745 ts_tablet_manager.cc:1428] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.054s user 0.002s sys 0.004s
I20250114 08:58:18.040264 21746 tablet_bootstrap.cc:492] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:18.040794 21746 ts_tablet_manager.cc:1397] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.083s user 0.018s sys 0.011s
I20250114 08:58:18.042176 21744 tablet_bootstrap.cc:492] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:18.042608 21744 ts_tablet_manager.cc:1397] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.002s
I20250114 08:58:18.046778 21745 tablet_bootstrap.cc:492] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:18.052282 21745 tablet_bootstrap.cc:654] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.056274 21868 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.065325 21746 raft_consensus.cc:357] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.073963 21746 raft_consensus.cc:383] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.074360 21746 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.073974 21868 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.076156 21868 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.076392 21746 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.072642 21744 raft_consensus.cc:357] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.077550 21744 raft_consensus.cc:383] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.077859 21744 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.078526 21744 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.079252 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.080345 21624 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:18.082513 21746 ts_tablet_manager.cc:1428] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20250114 08:58:18.083504 21746 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:18.083571 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.085070 21521 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:18.086165 21868 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.089167 21744 ts_tablet_manager.cc:1428] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.046s user 0.008s sys 0.000s
I20250114 08:58:18.092598 21744 tablet_bootstrap.cc:492] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:18.090201 21451 catalog_manager.cc:5526] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:18.098335 21699 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.099354 21746 tablet_bootstrap.cc:654] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.104799 21744 tablet_bootstrap.cc:654] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.110212 21962 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:18.113842 21745 tablet_bootstrap.cc:492] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:18.114512 21962 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.115038 21962 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.117249 21962 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.114395 21745 ts_tablet_manager.cc:1397] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.008s
I20250114 08:58:18.118742 21868 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:18.119016 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:18.120447 21596 leader_election.cc:343] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.120548 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:18.122103 21594 leader_election.cc:343] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.122529 21594 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:18.123605 21962 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:18.126376 21745 raft_consensus.cc:357] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.127187 21745 raft_consensus.cc:383] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.127527 21745 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.128376 21745 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.135329 21745 ts_tablet_manager.cc:1428] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.016s user 0.005s sys 0.003s
I20250114 08:58:18.136974 21745 tablet_bootstrap.cc:492] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:18.143338 21746 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:18.143865 21746 ts_tablet_manager.cc:1397] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.003s
I20250114 08:58:18.150841 21962 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:18.153307 21744 tablet_bootstrap.cc:492] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:18.154383 21744 ts_tablet_manager.cc:1397] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.005s
I20250114 08:58:18.157239 21744 raft_consensus.cc:357] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.158017 21744 raft_consensus.cc:383] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.158381 21744 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.159025 21744 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.161339 21744 ts_tablet_manager.cc:1428] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:58:18.162396 21744 tablet_bootstrap.cc:492] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:18.163481 21948 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.165058 21948 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.167460 21948 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.168027 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.168511 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.168787 21551 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:18.169337 21628 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:18.170308 21668 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:18.173131 21745 tablet_bootstrap.cc:654] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.177687 21771 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:18.178122 21771 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.178465 21771 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.183722 21746 raft_consensus.cc:357] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.184543 21746 raft_consensus.cc:383] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.184795 21746 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.185392 21746 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.192751 21746 ts_tablet_manager.cc:1428] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20250114 08:58:18.186095 21744 tablet_bootstrap.cc:654] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.195951 21746 tablet_bootstrap.cc:492] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:18.198755 21769 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.203747 21746 tablet_bootstrap.cc:654] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.203459 21769 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.205346 21769 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.208101 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.213315 21948 consensus_queue.cc:1035] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:18.212801 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.216017 21746 tablet_bootstrap.cc:492] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:18.216532 21746 ts_tablet_manager.cc:1397] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.003s
I20250114 08:58:18.222672 21746 raft_consensus.cc:357] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.223440 21746 raft_consensus.cc:383] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.223753 21746 raft_consensus.cc:738] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.224429 21746 consensus_queue.cc:260] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.224802 21596 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:18.230321 21771 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.233760 21745 tablet_bootstrap.cc:492] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:18.234266 21745 ts_tablet_manager.cc:1397] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.004s
I20250114 08:58:18.236675 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:18.236924 21745 raft_consensus.cc:357] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.237762 21551 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.237947 21745 raft_consensus.cc:383] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.238363 21745 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.239039 21745 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.240967 21745 ts_tablet_manager.cc:1428] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250114 08:58:18.243233 21745 tablet_bootstrap.cc:492] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:18.245543 21628 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:18.248417 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:18.250125 21771 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.251044 21551 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:18.253158 21668 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:18.253721 21746 ts_tablet_manager.cc:1428] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.037s user 0.002s sys 0.004s
I20250114 08:58:18.251154 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.254276 21629 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.254680 21746 tablet_bootstrap.cc:492] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:18.260810 21771 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:18.263295 21771 raft_consensus.cc:695] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:18.267536 21744 tablet_bootstrap.cc:492] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:18.268050 21744 ts_tablet_manager.cc:1397] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.009s
I20250114 08:58:18.270933 21629 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:18.271027 21744 raft_consensus.cc:357] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.264451 21771 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.273888 21744 raft_consensus.cc:383] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.274252 21744 raft_consensus.cc:738] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.274859 21744 consensus_queue.cc:260] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.277221 21744 ts_tablet_manager.cc:1428] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20250114 08:58:18.278141 21744 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:18.279618 22008 consensus_queue.cc:1035] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:18.282604 21746 tablet_bootstrap.cc:654] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.290470 21744 tablet_bootstrap.cc:654] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.299878 21744 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:18.300357 21744 ts_tablet_manager.cc:1397] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.003s
I20250114 08:58:18.304651 21744 raft_consensus.cc:357] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.305378 21744 raft_consensus.cc:383] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.305651 21744 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.307567 21746 tablet_bootstrap.cc:492] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:18.308034 21746 ts_tablet_manager.cc:1397] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.003s
I20250114 08:58:18.309237 21745 tablet_bootstrap.cc:654] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.311898 21769 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:18.310743 21746 raft_consensus.cc:357] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.317919 21746 raft_consensus.cc:383] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.311108 21744 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.318213 21746 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.318850 21746 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.320730 21764 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.323004 21764 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.326010 21744 ts_tablet_manager.cc:1428] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.023s user 0.005s sys 0.003s
I20250114 08:58:18.327313 21744 tablet_bootstrap.cc:492] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:18.329982 21764 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.324708 21962 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.331212 21962 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.334980 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.338070 21962 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.339530 21746 ts_tablet_manager.cc:1428] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.031s user 0.010s sys 0.001s
I20250114 08:58:18.339705 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:18.340117 21521 leader_election.cc:343] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.340514 21746 tablet_bootstrap.cc:492] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
W20250114 08:58:18.341322 21594 leader_election.cc:343] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.342792 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:18.349589 21596 leader_election.cc:343] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.350260 21596 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:18.351495 21769 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:18.349455 21868 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.352389 21868 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.353224 21699 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.348531 21452 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:18.361953 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.362641 21629 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:18.364569 21520 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.365370 21963 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:18.365741 21963 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.365826 21744 tablet_bootstrap.cc:654] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.366120 21963 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.371305 21963 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.359843 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.356456 21868 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.399266 21746 tablet_bootstrap.cc:654] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.380962 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.402254 21701 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:18.401263 21868 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.400955 21628 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:18.415256 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.417124 21629 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.413805 21960 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.419049 21960 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.419934 21745 tablet_bootstrap.cc:492] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:18.420399 21745 ts_tablet_manager.cc:1397] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.177s user 0.018s sys 0.022s
I20250114 08:58:18.422744 21520 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:18.420775 21962 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250114 08:58:18.430769 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:18.425513 21764 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:18.424618 21963 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.414873 21868 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.435292 21764 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.442714 21764 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.436164 21700 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.443956 21868 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.445257 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.445943 21554 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:18.449388 21594 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.451414 22052 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:18.451821 22052 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.452164 22052 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.452804 21960 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.454282 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.457959 21701 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:18.471753 21629 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:18.475051 21745 raft_consensus.cc:357] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.475876 21745 raft_consensus.cc:383] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.476145 21745 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.477041 21744 tablet_bootstrap.cc:492] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:18.476812 21745 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.477677 21744 ts_tablet_manager.cc:1397] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.151s user 0.028s sys 0.017s
I20250114 08:58:18.480777 21745 ts_tablet_manager.cc:1428] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.060s user 0.009s sys 0.000s
I20250114 08:58:18.481269 21744 raft_consensus.cc:357] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.493155 21744 raft_consensus.cc:383] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.493211 21962 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.481802 21745 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:18.494037 21962 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.495149 21769 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:18.496012 21962 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.493708 21744 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.484210 21700 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
W20250114 08:58:18.498464 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 68712.6us
I20250114 08:58:18.501951 21520 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.503299 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.503053 21744 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.503940 21555 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:18.515177 21744 ts_tablet_manager.cc:1428] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.037s user 0.006s sys 0.001s
I20250114 08:58:18.504108 21868 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:18.549713 21868 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:18.550686 21868 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.554136 21744 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:18.557006 21746 tablet_bootstrap.cc:492] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:18.558632 21746 ts_tablet_manager.cc:1397] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.218s user 0.024s sys 0.041s
I20250114 08:58:18.573719 21745 tablet_bootstrap.cc:654] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.569167 21963 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.577944 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.519920 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.575021 21866 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.582193 22056 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:18.576331 21960 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250114 08:58:18.582432 21963 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.606921 21963 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.584805 21866 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.580108 21764 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.598749 22056 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.609407 22056 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.599537 22008 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:18.610466 22052 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.614764 22008 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.618582 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.626026 21624 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:18.627075 22008 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.628985 21866 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.629063 21746 raft_consensus.cc:357] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.629897 21746 raft_consensus.cc:383] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.630203 21746 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.630841 21746 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.633077 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.633824 21551 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:18.635504 22056 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.637275 21520 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.638690 21746 ts_tablet_manager.cc:1428] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.080s user 0.006s sys 0.001s
I20250114 08:58:18.640592 22052 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.642762 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.643091 21764 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.643435 21697 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:18.612807 21700 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:18.650208 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:18.650897 21554 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.651543 21744 tablet_bootstrap.cc:654] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.620762 22061 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.641577 21963 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:18.654832 21963 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.655181 21963 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.652199 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.658172 21960 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.659178 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.659212 21960 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.659979 21624 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:18.661154 21628 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.657233 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.654296 22061 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.666060 21668 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:18.674798 22008 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:18.675187 22008 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:18.675562 22008 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:18.661173 22056 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.678879 21745 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:18.679255 21745 ts_tablet_manager.cc:1397] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.198s user 0.010s sys 0.016s
I20250114 08:58:18.682184 21960 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.678283 21950 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250114 08:58:18.682546 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.683444 21627 raft_consensus.cc:2388] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:18.685402 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:18.688315 21746 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:18.688585 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.678328 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:18.691682 21745 raft_consensus.cc:357] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.692757 21745 raft_consensus.cc:383] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.692950 21745 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.693656 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.694783 21963 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:18.695219 21596 leader_election.cc:343] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.672297 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.696076 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.696753 21624 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:18.698118 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.698760 21627 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:18.698374 21594 leader_election.cc:343] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.699291 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:18.699909 21699 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.680858 21555 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:18.699276 21594 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:18.705191 21520 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.710245 21962 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:18.705843 21629 raft_consensus.cc:2388] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:18.722182 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:18.717550 21764 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:18.719281 21668 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:18.731277 21764 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.731593 22057 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.731750 21764 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.732088 22057 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.733956 22064 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:18.734267 22057 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.734382 22064 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.734748 22064 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.730796 21745 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.750170 21744 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:18.752379 21744 ts_tablet_manager.cc:1397] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.198s user 0.005s sys 0.012s
I20250114 08:58:18.756183 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.765466 21628 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
W20250114 08:58:18.767925 21520 leader_election.cc:343] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.723593 22061 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.768432 21452 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:18.770464 21520 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.781699 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.782408 21696 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:18.782645 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.770622 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:18.785738 21771 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:18.785993 21554 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:18.786260 21771 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.783334 21698 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:18.752996 22056 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 08:58:18.788247 21771 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.752852 21963 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.762979 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:18.794730 21594 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.735575 21627 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:18.792124 22059 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:18.799391 21745 ts_tablet_manager.cc:1428] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.120s user 0.008s sys 0.001s
I20250114 08:58:18.801851 21745 tablet_bootstrap.cc:492] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:18.802237 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:18.803047 21624 raft_consensus.cc:2388] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:18.767165 22008 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.804208 21520 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.797425 22056 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:18.814838 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:18.805361 22059 raft_consensus.cc:695] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:18.808387 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.799820 21521 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:18.822180 22057 raft_consensus.cc:2758] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250114 08:58:18.815347 22056 raft_consensus.cc:695] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:18.824620 21701 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:18.824005 22056 consensus_queue.cc:237] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.830364 21699 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:18.813915 22061 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:18.845866 22061 raft_consensus.cc:695] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:18.845253 22059 consensus_queue.cc:237] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.846748 22061 consensus_queue.cc:237] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.849237 21764 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.851269 21764 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:18.852857 21771 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:18.856662 21555 raft_consensus.cc:2363] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:18.869971 22008 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.874380 21668 leader_election.cc:400] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:18.874186 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.874699 21668 leader_election.cc:403] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:58:18.875020 21627 raft_consensus.cc:2363] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:18.875931 21669 leader_election.cc:400] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:18.876209 21669 leader_election.cc:403] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:18.876835 21771 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.879030 21744 raft_consensus.cc:357] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.879745 21744 raft_consensus.cc:383] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:18.880126 21744 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:18.882640 22008 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.880951 21744 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.888314 21744 ts_tablet_manager.cc:1428] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.136s user 0.008s sys 0.000s
I20250114 08:58:18.889215 21746 tablet_bootstrap.cc:654] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.889402 21744 tablet_bootstrap.cc:492] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:18.892609 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:18.892860 21953 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:18.896330 22064 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.899333 22008 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:18.885035 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.916095 21628 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:18.922435 21453 catalog_manager.cc:5526] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:18.925884 22064 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:18.914007 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 92188.8us
I20250114 08:58:18.930635 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:18.940264 21554 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.919957 21771 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:18.951746 21745 tablet_bootstrap.cc:654] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:18.937376 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:18.953892 21624 raft_consensus.cc:2388] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:18.957542 21744 tablet_bootstrap.cc:654] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:18.958225 21521 leader_election.cc:343] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:18.959668 21521 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:18.960537 22057 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:18.960899 22057 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:18.961242 22057 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:18.969365 21452 catalog_manager.cc:5526] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:18.974400 21746 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:18.974911 21746 ts_tablet_manager.cc:1397] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.287s user 0.020s sys 0.005s
I20250114 08:58:18.988204 22057 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.991432 21866 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:18.991946 21866 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.975368 21628 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:18.995540 21520 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:18.995752 21866 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:18.996707 21963 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:18.997175 21963 raft_consensus.cc:695] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:18.997946 21963 consensus_queue.cc:237] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:18.998500 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:18.999148 21551 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:19.004619 21745 tablet_bootstrap.cc:492] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:19.005461 21745 ts_tablet_manager.cc:1397] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.205s user 0.019s sys 0.016s
I20250114 08:58:18.998512 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:19.007457 21668 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:19.010054 21745 raft_consensus.cc:357] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.008451 21746 raft_consensus.cc:357] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.010809 21745 raft_consensus.cc:383] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.011169 21745 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.010886 21746 raft_consensus.cc:383] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.011736 21746 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.011803 21745 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.012521 21746 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.013506 21629 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:19.014148 21745 ts_tablet_manager.cc:1428] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250114 08:58:19.014776 21866 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:19.015312 21866 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:19.015674 21866 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.016119 21745 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:19.025389 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:19.029752 22057 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.033376 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:19.034238 21701 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.034520 21700 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:19.038149 21554 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:19.040711 21668 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:19.045332 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:19.045542 22008 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:19.046586 22008 raft_consensus.cc:695] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:19.047441 22008 consensus_queue.cc:237] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:19.049650 21520 leader_election.cc:343] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.053525 21744 tablet_bootstrap.cc:492] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:19.054075 21744 ts_tablet_manager.cc:1397] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.165s user 0.018s sys 0.000s
I20250114 08:58:19.057540 21746 ts_tablet_manager.cc:1428] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.082s user 0.009s sys 0.000s
I20250114 08:58:19.059360 21746 tablet_bootstrap.cc:492] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:19.050282 21866 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.061333 21701 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:19.074887 21453 catalog_manager.cc:5526] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:19.080078 21629 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:19.079679 21453 catalog_manager.cc:5526] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:19.083035 21963 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:19.083233 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:19.084729 21630 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.084048 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:19.088826 21555 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.090425 21521 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:19.094506 22071 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:19.094996 22071 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:19.096987 22071 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.098897 21744 raft_consensus.cc:357] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.099613 21744 raft_consensus.cc:383] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.099874 21744 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.078619 21866 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:19.122467 21746 tablet_bootstrap.cc:654] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.122350 21630 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:19.125485 21744 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.128608 21452 catalog_manager.cc:5526] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:19.129940 21744 ts_tablet_manager.cc:1428] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.074s user 0.007s sys 0.000s
I20250114 08:58:19.130893 21744 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:19.119218 21555 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:19.133497 21668 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:19.135432 21866 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:19.135907 21866 raft_consensus.cc:695] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:19.136762 21866 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.142072 21700 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:19.143882 22057 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:19.151840 21745 tablet_bootstrap.cc:654] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.160423 21451 catalog_manager.cc:5526] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:19.181394 21746 tablet_bootstrap.cc:492] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:19.181974 21746 ts_tablet_manager.cc:1397] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.123s user 0.020s sys 0.001s
I20250114 08:58:19.189224 21744 tablet_bootstrap.cc:654] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.189255 21746 raft_consensus.cc:357] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.190336 21746 raft_consensus.cc:383] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.190608 21746 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.210788 22071 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:19.211701 22071 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.211135 21745 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:19.215781 21745 ts_tablet_manager.cc:1397] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.200s user 0.014s sys 0.003s
I20250114 08:58:19.216409 21451 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:19.218351 21745 raft_consensus.cc:357] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.219070 21745 raft_consensus.cc:383] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.219339 21745 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.220881 22071 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.222334 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:19.224928 21520 leader_election.cc:343] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.219928 21745 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.222882 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:19.228214 21521 leader_election.cc:343] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.228624 21745 ts_tablet_manager.cc:1428] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20250114 08:58:19.228714 21521 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:19.229599 21745 tablet_bootstrap.cc:492] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:19.229787 22057 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:19.209622 21746 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.236163 21746 ts_tablet_manager.cc:1428] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.054s user 0.011s sys 0.002s
I20250114 08:58:19.237681 21746 tablet_bootstrap.cc:492] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:19.244764 21745 tablet_bootstrap.cc:654] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.259168 21744 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:19.259816 21744 ts_tablet_manager.cc:1397] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.129s user 0.006s sys 0.014s
I20250114 08:58:19.266918 21745 tablet_bootstrap.cc:492] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:19.267426 21745 ts_tablet_manager.cc:1397] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.019s
I20250114 08:58:19.270411 21745 raft_consensus.cc:357] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.271118 21745 raft_consensus.cc:383] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.271399 21745 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.271986 21745 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.273814 21746 tablet_bootstrap.cc:654] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.274132 21745 ts_tablet_manager.cc:1428] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 08:58:19.275027 21745 tablet_bootstrap.cc:492] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:19.263094 21744 raft_consensus.cc:357] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.275426 21744 raft_consensus.cc:383] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.275698 21744 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.276306 21744 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.278568 21744 ts_tablet_manager.cc:1428] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.018s user 0.003s sys 0.004s
I20250114 08:58:19.279425 21744 tablet_bootstrap.cc:492] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:19.285171 21744 tablet_bootstrap.cc:654] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.288728 21745 tablet_bootstrap.cc:654] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.302023 21746 tablet_bootstrap.cc:492] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:19.302498 21746 ts_tablet_manager.cc:1397] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.003s
I20250114 08:58:19.304383 22052 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:19.309128 21700 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.309700 21746 raft_consensus.cc:357] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.310559 21746 raft_consensus.cc:383] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.311993 21746 raft_consensus.cc:738] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.312783 21746 consensus_queue.cc:260] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.315651 22057 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:19.317179 22057 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.319152 22057 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.321012 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:19.322782 21521 leader_election.cc:343] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.324218 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:19.325102 21628 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:19.326381 21520 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:19.328946 22071 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:19.329314 22071 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:19.329633 22071 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.333822 21963 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 08:58:19.337692 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:19.344408 22071 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.358330 21963 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20250114 08:58:19.368012 21960 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20250114 08:58:19.396525 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:19.387638 22055 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:19.398047 21521 leader_election.cc:343] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.398257 22055 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.388128 21628 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:19.401968 22055 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.405478 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:19.406630 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:19.407253 21628 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.411067 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:19.412431 21594 leader_election.cc:343] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.414258 22061 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 08:58:19.417434 21596 leader_election.cc:343] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.417845 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:19.418552 21950 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:19.425216 22061 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:19.425909 22061 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.429360 22061 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.430491 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:19.431197 21629 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:19.430619 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:19.432250 21701 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:19.433151 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:19.435195 21950 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:19.435846 21950 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.437729 21628 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:19.438474 21950 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.439069 22061 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:19.439519 22061 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:19.439567 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:19.439996 22061 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:19.440820 21594 leader_election.cc:343] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.441078 21520 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:19.441305 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:19.441823 21963 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:19.442312 21963 raft_consensus.cc:695] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:19.443140 21963 consensus_queue.cc:237] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:19.450510 21596 leader_election.cc:343] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:19.450997 21596 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:19.455662 22055 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:19.466143 21701 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:19.389547 21746 ts_tablet_manager.cc:1428] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.087s user 0.008s sys 0.000s
I20250114 08:58:19.475955 21746 tablet_bootstrap.cc:492] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:19.393013 21630 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:19.478112 22057 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:19.488982 22071 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.491186 22061 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.493480 22064 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:19.495304 22061 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.496728 22064 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:19.499414 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:19.499979 21699 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.497188 21744 tablet_bootstrap.cc:492] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:19.505749 21744 ts_tablet_manager.cc:1397] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.226s user 0.023s sys 0.019s
I20250114 08:58:19.507892 21699 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:19.510313 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:19.511880 22057 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:19.512372 22057 raft_consensus.cc:695] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:19.512578 22059 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:19.513195 22057 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.518445 21745 tablet_bootstrap.cc:492] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:19.520627 21745 ts_tablet_manager.cc:1397] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.246s user 0.025s sys 0.081s
I20250114 08:58:19.534480 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:19.535228 21625 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.559939 22059 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:19.580683 21625 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:19.508423 21744 raft_consensus.cc:357] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.600615 21744 raft_consensus.cc:383] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.600966 21744 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.618664 21746 tablet_bootstrap.cc:654] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.611769 21745 raft_consensus.cc:357] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.633035 21745 raft_consensus.cc:383] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.633368 21745 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.624640 21451 catalog_manager.cc:5526] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:19.636041 21745 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.629454 21700 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:19.642174 22070 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:19.644773 21866 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 08:58:19.662824 21520 consensus_peers.cc:487] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Couldn't send request to peer d7732324fb5f48d3b16a29d4bb2584ed. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:19.699994 21745 ts_tablet_manager.cc:1428] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.179s user 0.006s sys 0.000s
I20250114 08:58:19.702811 21745 tablet_bootstrap.cc:492] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:19.725769 21744 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.733376 22104 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:19.762861 22104 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.765502 22104 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.765852 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:19.766559 21625 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:19.767839 21520 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:19.770733 22103 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:19.771006 21746 tablet_bootstrap.cc:492] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:19.771083 22103 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:19.771663 22103 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.771703 21746 ts_tablet_manager.cc:1397] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.296s user 0.018s sys 0.012s
I20250114 08:58:19.774811 21746 raft_consensus.cc:357] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.775518 21746 raft_consensus.cc:383] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:19.775779 21746 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:19.776425 21746 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.762099 21745 tablet_bootstrap.cc:654] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.779021 21746 ts_tablet_manager.cc:1428] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250114 08:58:19.779672 22103 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:19.781551 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 125649us
I20250114 08:58:19.779937 21746 tablet_bootstrap.cc:492] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:19.803929 21451 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:19.842738 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:19.843416 21625 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:19.846905 21744 ts_tablet_manager.cc:1428] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.341s user 0.010s sys 0.005s
I20250114 08:58:19.848001 21744 tablet_bootstrap.cc:492] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:19.848811 22103 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:19.861764 22114 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250114 08:58:19.918675 21746 tablet_bootstrap.cc:654] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:19.925365 21625 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:19.926730 21520 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:19.928122 22104 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:19.928714 22104 raft_consensus.cc:695] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:19.931450 22104 consensus_queue.cc:237] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:19.932780 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:19.933665 21698 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:19.938498 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 76728.5us
I20250114 08:58:19.938517 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:19.944634 21698 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:19.958343 21745 tablet_bootstrap.cc:492] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:19.958854 21745 ts_tablet_manager.cc:1397] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.256s user 0.012s sys 0.006s
I20250114 08:58:19.984277 21744 tablet_bootstrap.cc:654] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.001626 21745 raft_consensus.cc:357] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.002630 21745 raft_consensus.cc:383] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.002981 21745 raft_consensus.cc:738] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.003911 21745 consensus_queue.cc:260] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.012856 21745 ts_tablet_manager.cc:1428] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.054s user 0.009s sys 0.000s
I20250114 08:58:20.013831 21745 tablet_bootstrap.cc:492] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:20.014442 21452 catalog_manager.cc:5526] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:20.021255 22104 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:20.023109 22104 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.031502 21746 tablet_bootstrap.cc:492] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:20.036721 21746 ts_tablet_manager.cc:1397] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.257s user 0.023s sys 0.024s
I20250114 08:58:20.025830 22104 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.046111 22103 consensus_queue.cc:1035] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:20.049957 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:20.051924 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:20.052227 21521 consensus_peers.cc:487] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Couldn't send request to peer cec0ef3bbb654988bf2123d8eef7a0a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:20.052773 21625 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
W20250114 08:58:20.054534 21521 leader_election.cc:343] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.054399 21746 raft_consensus.cc:357] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.055141 21746 raft_consensus.cc:383] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.055487 21746 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.056118 21746 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.046418 22104 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.060809 21745 tablet_bootstrap.cc:654] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:20.061179 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 63071.2us
I20250114 08:58:20.061489 21746 ts_tablet_manager.cc:1428] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20250114 08:58:20.062410 21746 tablet_bootstrap.cc:492] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:20.067965 21746 tablet_bootstrap.cc:654] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.068955 21950 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.069633 21950 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.072247 21950 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.076525 22070 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:20.113587 21520 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:20.114766 22104 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:20.115162 21744 tablet_bootstrap.cc:492] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:20.115691 21744 ts_tablet_manager.cc:1397] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.268s user 0.006s sys 0.018s
I20250114 08:58:20.115129 22104 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:20.117317 22104 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.119906 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:20.129729 21744 raft_consensus.cc:357] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.133450 21744 raft_consensus.cc:383] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.133853 21744 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
W20250114 08:58:20.134033 21596 leader_election.cc:343] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.134670 21744 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.138432 21744 ts_tablet_manager.cc:1428] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20250114 08:58:20.143728 21744 tablet_bootstrap.cc:492] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:20.139884 22104 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.155411 22104 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.155135 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:20.158918 21552 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:20.145526 21745 tablet_bootstrap.cc:492] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:20.161293 21745 ts_tablet_manager.cc:1397] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.148s user 0.014s sys 0.025s
I20250114 08:58:20.156761 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:20.163226 21521 leader_election.cc:343] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.172348 21950 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.163935 21745 raft_consensus.cc:357] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.173271 21745 raft_consensus.cc:383] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.172941 21950 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.167793 21744 tablet_bootstrap.cc:654] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.174882 21950 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.173555 21745 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.176597 21745 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.177451 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:20.177891 21594 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:20.178134 21630 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.178602 22119 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:20.179253 22119 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:20.179276 21745 ts_tablet_manager.cc:1428] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250114 08:58:20.179605 22119 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.186693 22119 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.187268 21746 tablet_bootstrap.cc:492] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:20.188158 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:20.189447 21630 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:20.189945 21745 tablet_bootstrap.cc:492] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:20.192929 21746 ts_tablet_manager.cc:1397] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.131s user 0.008s sys 0.005s
I20250114 08:58:20.193018 22119 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.195264 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:20.195593 21745 tablet_bootstrap.cc:654] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.196166 21698 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.195333 21746 raft_consensus.cc:357] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.199864 21746 raft_consensus.cc:383] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.200196 21746 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.200963 21746 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.202224 21744 tablet_bootstrap.cc:492] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:20.202700 21744 ts_tablet_manager.cc:1397] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.004s
I20250114 08:58:20.203631 21698 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:20.204804 21746 ts_tablet_manager.cc:1428] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20250114 08:58:20.205823 21746 tablet_bootstrap.cc:492] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:20.206818 21745 tablet_bootstrap.cc:492] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:20.207260 21745 ts_tablet_manager.cc:1397] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20250114 08:58:20.254164 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:20.257082 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:20.258107 21551 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.274709 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:20.281337 22134 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:20.272346 21520 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:20.299873 21596 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:20.309159 22134 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:20.309229 21744 raft_consensus.cc:357] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.310699 21744 raft_consensus.cc:383] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.311050 21744 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.312327 21744 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.312656 21551 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:20.310045 22134 raft_consensus.cc:695] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:20.332805 22059 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:20.336668 21744 ts_tablet_manager.cc:1428] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.134s user 0.010s sys 0.001s
I20250114 08:58:20.341305 21744 tablet_bootstrap.cc:492] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:20.345347 21630 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.356139 21745 raft_consensus.cc:357] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.360073 22059 raft_consensus.cc:695] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:58:20.360616 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 45875.4us
I20250114 08:58:20.365976 21745 raft_consensus.cc:383] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.370000 21745 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.367537 22134 consensus_queue.cc:237] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.372290 21745 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.383397 22059 consensus_queue.cc:237] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.384151 21745 ts_tablet_manager.cc:1428] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.177s user 0.009s sys 0.000s
I20250114 08:58:20.403041 21868 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 08:58:20.415055 21745 tablet_bootstrap.cc:492] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
W20250114 08:58:20.425949 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:20.441787 21744 tablet_bootstrap.cc:654] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.517797 21746 tablet_bootstrap.cc:654] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.591490 22059 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.592139 22059 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.609997 21744 tablet_bootstrap.cc:492] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:20.609285 21452 catalog_manager.cc:5526] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:20.610620 21744 ts_tablet_manager.cc:1397] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.270s user 0.007s sys 0.036s
I20250114 08:58:20.616492 21745 tablet_bootstrap.cc:654] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.625157 22112 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.625778 22112 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.627837 22112 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:20.629858 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:20.637514 21669 leader_election.cc:343] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.637368 21744 raft_consensus.cc:357] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.639907 21744 raft_consensus.cc:383] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.640192 21744 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.651402 21453 catalog_manager.cc:5526] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:20.656482 22138 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.653143 22055 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.641866 21744 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.658645 22055 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.658828 22103 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:20.659739 22138 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.660235 22059 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.661715 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:20.662341 22055 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.662412 21553 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:20.663442 22064 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.664132 21594 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:20.664476 21745 tablet_bootstrap.cc:492] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:20.663909 22064 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.664655 22138 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.664914 22055 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:20.664754 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:20.665341 22055 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:20.665699 22055 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:20.666244 21596 leader_election.cc:343] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.666316 22064 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:20.667085 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:20.668138 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:20.668373 21521 leader_election.cc:343] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
W20250114 08:58:20.669495 21669 leader_election.cc:343] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.669806 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:20.670532 21704 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:20.671923 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:20.672753 21745 ts_tablet_manager.cc:1397] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.258s user 0.040s sys 0.022s
I20250114 08:58:20.674717 22055 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.680420 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:20.683233 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:20.690519 21551 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
W20250114 08:58:20.685984 21668 leader_election.cc:343] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.680191 21521 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:20.679236 21745 raft_consensus.cc:357] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.691756 21668 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:20.685211 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:20.693923 22103 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:20.695622 21625 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:20.695935 22103 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:20.696343 22103 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.698323 22118 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 08:58:20.701351 21520 leader_election.cc:343] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.701709 21668 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:20.689218 21744 ts_tablet_manager.cc:1428] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.073s user 0.005s sys 0.002s
I20250114 08:58:20.706277 22064 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:20.707458 22064 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:20.707813 22064 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.713977 21520 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:20.714244 22141 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20250114 08:58:20.694618 21745 raft_consensus.cc:383] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.723065 21745 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.724874 22138 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:20.728518 22055 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.729501 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:20.734463 21744 tablet_bootstrap.cc:492] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:20.734242 21745 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.738474 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:20.739789 21596 leader_election.cc:343] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.743562 21745 ts_tablet_manager.cc:1428] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20250114 08:58:20.749547 21745 tablet_bootstrap.cc:492] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:20.743227 22103 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:20.771721 21521 consensus_peers.cc:487] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Couldn't send request to peer cec0ef3bbb654988bf2123d8eef7a0a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:20.775903 22064 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.783178 22103 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:20.784127 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:20.784066 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:20.787369 21698 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
W20250114 08:58:20.792030 21520 leader_election.cc:343] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.792542 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:20.793742 22135 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 08:58:20.795856 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:20.799218 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:20.799661 21746 tablet_bootstrap.cc:492] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
W20250114 08:58:20.799715 21669 leader_election.cc:343] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:20.800038 21553 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:20.800259 21746 ts_tablet_manager.cc:1397] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.595s user 0.052s sys 0.051s
I20250114 08:58:20.801730 21668 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:20.803679 22112 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 08:58:20.795303 22064 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:20.810606 21745 tablet_bootstrap.cc:654] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.810902 22135 consensus_queue.cc:1035] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 08:58:20.813979 21744 tablet_bootstrap.cc:654] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.811596 21746 raft_consensus.cc:357] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.819556 22138 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:20.819885 21746 raft_consensus.cc:383] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.820345 21746 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.821091 21746 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.824982 21746 ts_tablet_manager.cc:1428] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20250114 08:58:20.830535 21551 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:20.832183 21746 tablet_bootstrap.cc:492] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:20.838413 21594 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:20.841938 21960 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:20.843369 21960 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:20.846432 21960 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.858090 21746 tablet_bootstrap.cc:654] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.858817 21745 tablet_bootstrap.cc:492] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:20.859352 21745 ts_tablet_manager.cc:1397] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.009s
I20250114 08:58:20.862675 21745 raft_consensus.cc:357] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.863649 21745 raft_consensus.cc:383] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.863951 21745 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.864773 21745 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.867210 21745 ts_tablet_manager.cc:1428] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250114 08:58:20.876271 21745 tablet_bootstrap.cc:492] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:20.876519 21744 tablet_bootstrap.cc:492] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:20.877023 21744 ts_tablet_manager.cc:1397] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.164s user 0.012s sys 0.003s
I20250114 08:58:20.879547 21744 raft_consensus.cc:357] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.880223 21744 raft_consensus.cc:383] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.880528 21744 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.881310 21744 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.885524 21453 catalog_manager.cc:5526] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:20.897656 21745 tablet_bootstrap.cc:654] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.903959 21746 tablet_bootstrap.cc:492] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:20.904387 21746 ts_tablet_manager.cc:1397] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.073s user 0.025s sys 0.019s
I20250114 08:58:20.906749 21746 raft_consensus.cc:357] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.907383 21746 raft_consensus.cc:383] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.907433 21744 ts_tablet_manager.cc:1428] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250114 08:58:20.907647 21746 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.908227 21746 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.910526 21746 ts_tablet_manager.cc:1428] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 08:58:20.911495 21746 tablet_bootstrap.cc:492] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:20.915441 21744 tablet_bootstrap.cc:492] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:20.917953 21746 tablet_bootstrap.cc:654] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.925544 22055 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:20.930094 21746 tablet_bootstrap.cc:492] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:20.930603 21746 ts_tablet_manager.cc:1397] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250114 08:58:20.932760 21745 tablet_bootstrap.cc:492] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:20.933737 21745 ts_tablet_manager.cc:1397] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.004s
I20250114 08:58:20.932801 21744 tablet_bootstrap.cc:654] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.933730 21746 raft_consensus.cc:357] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.934634 21746 raft_consensus.cc:383] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.934911 21746 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.935619 21746 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.936355 21745 raft_consensus.cc:357] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.937178 21745 raft_consensus.cc:383] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.937433 21745 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.938097 21746 ts_tablet_manager.cc:1428] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250114 08:58:20.938076 21745 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.940125 21745 ts_tablet_manager.cc:1428] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 08:58:20.940725 21746 tablet_bootstrap.cc:492] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:20.941952 21745 tablet_bootstrap.cc:492] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:20.947381 21746 tablet_bootstrap.cc:654] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.948380 21745 tablet_bootstrap.cc:654] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:20.955986 21744 tablet_bootstrap.cc:492] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:20.956508 21744 ts_tablet_manager.cc:1397] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.004s
I20250114 08:58:20.960660 21960 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:20.961683 21746 tablet_bootstrap.cc:492] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:20.962167 21746 ts_tablet_manager.cc:1397] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.012s
I20250114 08:58:20.964920 21746 raft_consensus.cc:357] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.965606 21746 raft_consensus.cc:383] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.965870 21746 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.966491 21746 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.970422 21746 ts_tablet_manager.cc:1428] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250114 08:58:20.961263 21744 raft_consensus.cc:357] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.972652 21744 raft_consensus.cc:383] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:20.972959 21744 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:20.973716 21744 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.976014 21744 ts_tablet_manager.cc:1428] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.019s user 0.003s sys 0.001s
I20250114 08:58:20.977098 21744 tablet_bootstrap.cc:492] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:20.979521 21745 tablet_bootstrap.cc:492] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:20.980084 21745 ts_tablet_manager.cc:1397] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.004s
I20250114 08:58:20.982519 21746 tablet_bootstrap.cc:492] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:20.990780 22118 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:20.991371 22118 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:20.995227 22118 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:20.998271 21744 tablet_bootstrap.cc:654] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.000748 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.001596 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:21.011443 21668 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:21.012537 22118 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.014570 21746 tablet_bootstrap.cc:654] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.019869 21745 raft_consensus.cc:357] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.020680 21745 raft_consensus.cc:383] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.020973 21745 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.021582 21745 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.023828 21745 ts_tablet_manager.cc:1428] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.043s user 0.001s sys 0.002s
I20250114 08:58:21.029795 21704 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.035657 22103 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.036198 22103 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.038281 22103 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.039548 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.040272 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.040987 21698 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:21.040215 21630 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:21.043179 22136 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.043597 21520 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:21.044673 22135 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:21.045022 22135 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:21.045349 22135 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.059263 21744 tablet_bootstrap.cc:492] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.059840 21744 ts_tablet_manager.cc:1397] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.014s
I20250114 08:58:21.043632 22136 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.064035 22136 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.066946 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:21.069154 21521 leader_election.cc:343] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.066844 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.069851 21625 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:21.076858 21745 tablet_bootstrap.cc:492] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:21.077176 22135 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.080731 21520 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:21.097978 22136 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:21.105542 21744 raft_consensus.cc:357] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.106235 21744 raft_consensus.cc:383] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.106495 21744 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.098420 22136 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:21.108366 21744 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.108186 22136 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.121609 21744 ts_tablet_manager.cc:1428] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.061s user 0.012s sys 0.008s
I20250114 08:58:21.123821 21746 tablet_bootstrap.cc:492] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:21.124409 21746 ts_tablet_manager.cc:1397] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.142s user 0.024s sys 0.009s
I20250114 08:58:21.125006 21744 tablet_bootstrap.cc:492] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:21.127287 22136 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.131024 21745 tablet_bootstrap.cc:654] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.131803 21746 raft_consensus.cc:357] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.133268 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:21.133652 21746 raft_consensus.cc:383] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.133950 21746 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.133970 21698 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.134130 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:21.133320 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:21.134938 21626 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.134689 21746 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.135105 21630 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.137584 22136 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.130653 22135 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.140938 21744 tablet_bootstrap.cc:654] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.142488 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:21.145251 21521 leader_election.cc:343] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.147729 21746 ts_tablet_manager.cc:1428] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20250114 08:58:21.157747 21746 tablet_bootstrap.cc:492] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:21.160372 21698 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:21.161607 21521 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:21.169306 22118 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.172575 22118 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.174551 22118 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:21.176364 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.177088 21624 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:21.177276 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:21.166504 22138 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:21.178689 21669 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:21.177984 21551 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:21.186179 22138 raft_consensus.cc:695] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:21.186661 21746 tablet_bootstrap.cc:654] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.187326 22138 consensus_queue.cc:237] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.187862 22118 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:21.198146 22118 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:21.198792 22118 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.189558 21630 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:21.205675 21626 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:21.210176 21520 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:21.211503 22135 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:21.215085 22135 raft_consensus.cc:695] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:21.215991 22135 consensus_queue.cc:237] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.218200 21745 tablet_bootstrap.cc:492] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:21.218677 21745 ts_tablet_manager.cc:1397] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.142s user 0.018s sys 0.026s
I20250114 08:58:21.219519 22118 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.221449 21745 raft_consensus.cc:357] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.222184 21745 raft_consensus.cc:383] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.222469 21745 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.223486 21745 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.224798 21744 tablet_bootstrap.cc:492] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.225685 21744 ts_tablet_manager.cc:1397] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.102s user 0.010s sys 0.010s
I20250114 08:58:21.226526 21745 ts_tablet_manager.cc:1428] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250114 08:58:21.227656 21745 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:21.233497 21745 tablet_bootstrap.cc:654] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.234189 22059 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.234752 22059 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.236111 22118 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:21.237011 22059 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.238204 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.239107 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.239830 21746 tablet_bootstrap.cc:492] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
W20250114 08:58:21.239740 21521 leader_election.cc:343] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.240355 21746 ts_tablet_manager.cc:1397] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.011s
I20250114 08:58:21.236548 21744 raft_consensus.cc:357] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.240988 21744 raft_consensus.cc:383] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250114 08:58:21.242449 21520 leader_election.cc:343] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.243053 21520 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:21.244473 22136 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.245659 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:21.246016 21744 raft_consensus.cc:738] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.246500 21630 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.247109 21744 consensus_queue.cc:260] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.249392 21744 ts_tablet_manager.cc:1428] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250114 08:58:21.249413 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:21.250495 21553 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.265609 21451 catalog_manager.cc:5526] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:21.247579 21746 raft_consensus.cc:357] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.268007 21960 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.268713 21960 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.271661 21960 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.268034 21746 raft_consensus.cc:383] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.272364 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.275871 21746 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.272403 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:21.279381 21551 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:21.276180 21697 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 1: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:21.258148 21744 tablet_bootstrap.cc:492] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:21.281617 21596 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:21.282992 22055 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.301863 22055 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.302886 22118 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.303087 21744 tablet_bootstrap.cc:654] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.303417 22118 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.305446 22118 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:21.306913 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:21.307744 21551 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:21.308326 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.276958 21746 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.313664 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:21.315696 22168 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.317652 21745 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:21.318257 21745 ts_tablet_manager.cc:1397] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.091s user 0.029s sys 0.036s
I20250114 08:58:21.317592 21451 catalog_manager.cc:5526] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:21.321022 21745 raft_consensus.cc:357] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.326879 21745 raft_consensus.cc:383] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.327271 21745 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.327886 21745 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.329043 22168 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.324029 21744 tablet_bootstrap.cc:492] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.329900 21744 ts_tablet_manager.cc:1397] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.072s user 0.014s sys 0.003s
I20250114 08:58:21.324852 21746 ts_tablet_manager.cc:1428] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.084s user 0.007s sys 0.000s
I20250114 08:58:21.330417 21745 ts_tablet_manager.cc:1428] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250114 08:58:21.331655 21745 tablet_bootstrap.cc:492] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:21.332697 21744 raft_consensus.cc:357] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.333405 21744 raft_consensus.cc:383] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.333675 21744 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.334319 21744 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.338946 21746 tablet_bootstrap.cc:492] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:21.325424 21630 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:21.325832 21553 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:21.343343 21745 tablet_bootstrap.cc:654] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.344089 21744 ts_tablet_manager.cc:1428] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250114 08:58:21.344501 21668 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:21.345121 21744 tablet_bootstrap.cc:492] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:21.345959 22112 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:21.346451 22112 raft_consensus.cc:695] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:21.347254 22112 consensus_queue.cc:237] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.351001 21746 tablet_bootstrap.cc:654] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.358906 21744 tablet_bootstrap.cc:654] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.367264 21745 tablet_bootstrap.cc:492] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:21.367862 21745 ts_tablet_manager.cc:1397] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.000s
I20250114 08:58:21.368137 22055 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:21.370512 21950 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:21.370718 22168 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.371727 22168 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.374475 22168 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:21.374883 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.381198 21950 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.381814 21950 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.386382 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:21.393797 21744 tablet_bootstrap.cc:492] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.394268 21744 ts_tablet_manager.cc:1397] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.012s
I20250114 08:58:21.397799 21668 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:21.398587 22064 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.398711 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:21.401073 21594 leader_election.cc:343] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.376009 21745 raft_consensus.cc:357] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.402772 21745 raft_consensus.cc:383] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.403110 21745 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.405885 21950 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.406661 21745 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.407594 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:21.408964 21596 leader_election.cc:343] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.409004 21744 raft_consensus.cc:357] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.409446 21596 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:21.409751 21744 raft_consensus.cc:383] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.410166 22055 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.411237 21745 ts_tablet_manager.cc:1428] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.043s user 0.008s sys 0.002s
I20250114 08:58:21.412140 21746 tablet_bootstrap.cc:492] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:21.415813 21746 ts_tablet_manager.cc:1397] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.077s user 0.016s sys 0.005s
I20250114 08:58:21.410037 21744 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.424851 21744 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.426244 21451 catalog_manager.cc:5526] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:21.427490 21744 ts_tablet_manager.cc:1428] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.033s user 0.006s sys 0.002s
I20250114 08:58:21.428555 21744 tablet_bootstrap.cc:492] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:21.434634 21744 tablet_bootstrap.cc:654] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.455408 21745 tablet_bootstrap.cc:492] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:21.464808 21745 tablet_bootstrap.cc:654] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.420392 21746 raft_consensus.cc:357] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.538884 21746 raft_consensus.cc:383] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.539252 21746 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.539861 21746 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.596866 22135 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.597395 22135 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.599411 22135 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.602962 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.603734 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.607393 21746 ts_tablet_manager.cc:1428] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.191s user 0.007s sys 0.000s
W20250114 08:58:21.609329 21521 leader_election.cc:343] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.609489 21746 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
W20250114 08:58:21.622213 21520 leader_election.cc:343] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.622845 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:21.623845 22059 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.637889 21698 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.649892 22059 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:21.675153 21745 tablet_bootstrap.cc:492] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:21.675626 21745 ts_tablet_manager.cc:1397] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.243s user 0.037s sys 0.057s
I20250114 08:58:21.682351 21745 raft_consensus.cc:357] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.677505 21744 tablet_bootstrap.cc:492] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.679042 22135 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.684162 21744 ts_tablet_manager.cc:1397] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.256s user 0.053s sys 0.097s
I20250114 08:58:21.684108 22135 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.683137 21745 raft_consensus.cc:383] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.686319 21745 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.688340 21744 raft_consensus.cc:357] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.689216 21744 raft_consensus.cc:383] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.689502 21744 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.690176 21745 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.692636 21745 ts_tablet_manager.cc:1428] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.013s user 0.005s sys 0.002s
I20250114 08:58:21.693686 21745 tablet_bootstrap.cc:492] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:21.695142 22135 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.695058 21744 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.698297 21744 ts_tablet_manager.cc:1428] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250114 08:58:21.706827 21746 tablet_bootstrap.cc:654] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.707402 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.708599 21630 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 0.
I20250114 08:58:21.697990 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.710654 21744 tablet_bootstrap.cc:492] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
W20250114 08:58:21.711369 21521 leader_election.cc:343] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.712188 21520 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:21.715457 22135 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:21.716499 22055 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.717023 22055 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.718946 22055 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.722319 22059 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250114 08:58:21.720664 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.716524 22135 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:58:21.733625 21501 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 08:58:21.733772 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:21.734575 21704 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
W20250114 08:58:21.736539 21594 leader_election.cc:343] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.739025 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:21.730639 22135 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.742527 22055 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:21.746971 22055 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:21.747368 22055 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.750716 22136 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:21.752439 22136 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.751603 22135 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.755206 22136 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.756970 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:21.757293 22104 consensus_queue.cc:1035] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 08:58:21.759685 21521 leader_election.cc:343] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.762938 22135 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.766505 21744 tablet_bootstrap.cc:654] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.766913 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:21.767897 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:21.770828 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:21.773080 21625 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:21.776247 21697 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:21.774044 21521 leader_election.cc:343] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.773514 21745 tablet_bootstrap.cc:654] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
W20250114 08:58:21.772435 21520 leader_election.cc:343] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.781035 21520 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:21.786727 21625 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:21.787940 22055 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.790236 21520 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:21.791990 22136 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:21.792812 22136 raft_consensus.cc:695] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:21.793627 22136 consensus_queue.cc:237] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.791194 22135 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:21.795827 22055 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:21.802331 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:21.801919 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:21.803640 21699 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
W20250114 08:58:21.803813 21594 leader_election.cc:343] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Illegal state: must be running to vote when last-logged opid is not known
I20250114 08:58:21.807296 21745 tablet_bootstrap.cc:492] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:21.807826 21745 ts_tablet_manager.cc:1397] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.008s
I20250114 08:58:21.814174 21746 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:21.814692 21746 ts_tablet_manager.cc:1397] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.206s user 0.016s sys 0.004s
I20250114 08:58:21.811920 21745 raft_consensus.cc:357] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.815605 21745 raft_consensus.cc:383] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.815899 21745 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.817689 21745 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.819515 22104 consensus_queue.cc:1035] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:21.821790 21744 tablet_bootstrap.cc:492] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.822258 21744 ts_tablet_manager.cc:1397] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.112s user 0.005s sys 0.012s
I20250114 08:58:21.820423 21746 raft_consensus.cc:357] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.823726 21746 raft_consensus.cc:383] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.824018 21746 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.826270 21745 ts_tablet_manager.cc:1428] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20250114 08:58:21.835166 21745 tablet_bootstrap.cc:492] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Bootstrap starting.
I20250114 08:58:21.839399 21746 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.841848 21744 raft_consensus.cc:357] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.842649 21744 raft_consensus.cc:383] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.843020 21744 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:21.843822 21744 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.846211 21744 ts_tablet_manager.cc:1428] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20250114 08:58:21.893940 21744 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:21.894805 21746 ts_tablet_manager.cc:1428] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.080s user 0.009s sys 0.002s
I20250114 08:58:21.895874 21746 tablet_bootstrap.cc:492] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:21.897035 21453 catalog_manager.cc:5526] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:21.899564 21699 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:21.901786 21744 tablet_bootstrap.cc:654] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.911911 21745 tablet_bootstrap.cc:654] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.920557 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:21.922498 21950 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:21.922986 21950 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:21.923950 21950 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.949218 22168 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:21.982631 21746 tablet_bootstrap.cc:654] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:21.988372 21744 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:21.988955 21744 ts_tablet_manager.cc:1397] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.095s user 0.019s sys 0.041s
I20250114 08:58:21.999111 21744 raft_consensus.cc:357] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:21.999919 21744 raft_consensus.cc:383] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:21.999498 21453 catalog_manager.cc:5526] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:22.012183 21746 tablet_bootstrap.cc:492] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:22.000213 21744 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.008052 21745 tablet_bootstrap.cc:492] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: No bootstrap required, opened a new log
I20250114 08:58:22.008392 22176 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.012708 22196 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250114 08:58:22.018060 21744 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.019037 21745 ts_tablet_manager.cc:1397] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent bootstrapping tablet: real 0.184s user 0.011s sys 0.006s
I20250114 08:58:22.020637 21744 ts_tablet_manager.cc:1428] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.031s user 0.008s sys 0.004s
I20250114 08:58:22.021597 21745 raft_consensus.cc:357] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.022341 21745 raft_consensus.cc:383] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.022626 21745 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.023255 21745 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.021642 21744 tablet_bootstrap.cc:492] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:22.024628 22176 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.026580 22176 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.027247 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:22.027848 21746 ts_tablet_manager.cc:1397] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.005s
I20250114 08:58:22.027861 21551 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.029536 21668 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.030668 21744 tablet_bootstrap.cc:654] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:22.030512 21746 raft_consensus.cc:357] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.030974 22176 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:22.031690 22176 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:22.032116 22176 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.039163 22196 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.039804 22196 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.042279 22196 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.043848 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:22.044595 21551 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.045717 21668 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.054524 21744 tablet_bootstrap.cc:492] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:22.055008 21744 ts_tablet_manager.cc:1397] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.007s
I20250114 08:58:22.056037 22064 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:22.056448 22064 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:22.056833 22064 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.057677 21744 raft_consensus.cc:357] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.058393 21744 raft_consensus.cc:383] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.058636 21744 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.059408 21744 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.060935 22176 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.062196 21744 ts_tablet_manager.cc:1428] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250114 08:58:22.063228 21744 tablet_bootstrap.cc:492] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:22.063370 22176 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.070981 22064 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.071724 21744 tablet_bootstrap.cc:654] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:22.074501 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:22.075130 21555 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.078888 22064 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.031258 21746 raft_consensus.cc:383] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.080170 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:22.080765 21746 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.080976 21551 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.081467 21746 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.083622 21746 ts_tablet_manager.cc:1428] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.055s user 0.001s sys 0.003s
I20250114 08:58:22.084574 21746 tablet_bootstrap.cc:492] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:22.084977 21744 tablet_bootstrap.cc:492] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:22.085520 21744 ts_tablet_manager.cc:1397] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.010s
I20250114 08:58:22.051862 21745 ts_tablet_manager.cc:1428] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250114 08:58:22.088104 21744 raft_consensus.cc:357] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.088910 21744 raft_consensus.cc:383] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.089219 21744 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.089895 21744 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.090764 21746 tablet_bootstrap.cc:654] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:22.092795 21744 ts_tablet_manager.cc:1428] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250114 08:58:22.093814 21744 tablet_bootstrap.cc:492] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: Bootstrap starting.
I20250114 08:58:22.096724 21551 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.097954 21668 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.099660 21744 tablet_bootstrap.cc:654] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:22.106060 22112 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:22.106657 22112 raft_consensus.cc:695] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:22.107477 22112 consensus_queue.cc:237] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.110021 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:22.111287 21625 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.116407 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:22.117367 21630 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.119426 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:22.120205 21630 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.129659 21630 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.133872 21555 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.134006 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:22.134694 21630 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.138675 21668 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.139786 22112 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:22.150738 21746 tablet_bootstrap.cc:492] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:22.151723 21746 ts_tablet_manager.cc:1397] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.008s
I20250114 08:58:22.140445 22112 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:22.158955 21453 catalog_manager.cc:5526] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:22.159021 22112 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.174520 21746 raft_consensus.cc:357] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.175169 21746 raft_consensus.cc:383] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.175391 21746 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.175935 21746 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.177711 21746 ts_tablet_manager.cc:1428] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250114 08:58:22.178927 21704 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.180246 21746 tablet_bootstrap.cc:492] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Bootstrap starting.
I20250114 08:58:22.186914 21746 tablet_bootstrap.cc:654] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Neither blocks nor log segments found. Creating new log.
I20250114 08:58:22.191184 21630 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.193301 21744 tablet_bootstrap.cc:492] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: No bootstrap required, opened a new log
I20250114 08:58:22.193779 21744 ts_tablet_manager.cc:1397] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent bootstrapping tablet: real 0.100s user 0.020s sys 0.021s
I20250114 08:58:22.196350 21744 raft_consensus.cc:357] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.197126 21744 raft_consensus.cc:383] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.197396 21744 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.198086 21744 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.200021 22052 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.200551 22052 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.202093 21744 ts_tablet_manager.cc:1428] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250114 08:58:22.207878 22052 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:22.208503 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:22.212962 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:22.213444 21555 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:22.213641 21697 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 0.
I20250114 08:58:22.214991 21596 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:22.214938 22136 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.217792 22052 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:22.218194 22052 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:22.219223 22052 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.238544 22136 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.241961 21746 tablet_bootstrap.cc:492] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: No bootstrap required, opened a new log
I20250114 08:58:22.243499 21746 ts_tablet_manager.cc:1397] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.002s
I20250114 08:58:22.245816 22104 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.245467 21453 catalog_manager.cc:5526] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:22.246279 22104 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.248157 22104 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:22.248885 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:22.252552 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:22.253494 21704 raft_consensus.cc:2388] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 1: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:22.254719 21521 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:22.255033 21746 raft_consensus.cc:357] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.255767 21746 raft_consensus.cc:383] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 08:58:22.256021 21746 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Initialized, Role: FOLLOWER
I20250114 08:58:22.256062 22052 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.256878 22104 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.256786 21746 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.262892 21746 ts_tablet_manager.cc:1428] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250114 08:58:22.265925 22104 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 08:58:22.271348 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:22.271934 21555 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.271966 22052 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:22.273698 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:22.274356 21697 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.283581 21697 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:22.285027 21596 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:22.285629 22052 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:22.286878 22052 raft_consensus.cc:695] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:22.288079 22112 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.287691 22052 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.288595 22112 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.292997 21555 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:22.295212 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:22.295892 21551 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.295886 22112 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.298076 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.299592 22176 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:22.299947 22176 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:22.300267 22176 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.307564 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:22.322999 21625 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.333503 22176 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.336771 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:22.337371 21551 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.343559 21551 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.345227 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.346171 22196 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:22.346760 22196 raft_consensus.cc:695] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:22.347570 22196 consensus_queue.cc:237] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.350337 22176 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.351466 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:22.354351 21625 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.354351 21950 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:22.363320 21625 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.378326 21452 catalog_manager.cc:5526] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:22.395509 22052 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.398835 21453 catalog_manager.cc:5526] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:22.572261 22196 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.585337 22064 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.624197 22064 consensus_queue.cc:1035] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.630118 22064 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.630616 22064 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.633335 22196 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:22.633852 22196 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.638654 22112 consensus_queue.cc:1035] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:22.640698 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:22.641186 22196 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.641477 21551 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.641669 22064 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.645284 21668 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.646111 22064 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 08:58:22.646592 22064 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:22.647035 22064 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.648598 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:22.648836 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:22.649307 21625 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.649363 21628 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 0.
I20250114 08:58:22.655112 21669 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:22.656111 22176 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:22.656504 22176 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:22.656831 22176 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:22.646600 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:22.658773 21551 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.661711 22064 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.663475 22176 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.667243 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:22.667851 21630 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:22.671602 22176 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.678607 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:22.679288 21554 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:22.679221 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:22.679883 21553 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.682135 22064 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:22.686093 21554 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:22.688761 21630 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:22.691906 21669 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:22.693835 22176 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:22.689311 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:22.696808 22176 raft_consensus.cc:695] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:22.696954 21628 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 0 FOLLOWER]: Advancing to term 1
I20250114 08:58:22.697695 22176 consensus_queue.cc:237] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.701486 21553 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.709949 21668 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:22.711140 22064 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 08:58:22.712625 22064 raft_consensus.cc:695] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:22.713547 22064 consensus_queue.cc:237] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:22.716208 21628 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:22.736362 21452 catalog_manager.cc:5526] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 2, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:22.759344 22055 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.778291 21960 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.785404 21452 catalog_manager.cc:5526] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:22.839304 22064 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:22.871029 22217 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:23.131029 21630 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 08:58:23.132419 22217 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:23.148726 21554 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 08:58:23.150300 22112 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:23.172420 22217 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 08:58:23.209055 22176 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 08:58:23.625749 20382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 08:58:23.659963 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 08:58:23.664618 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:58:23.684898 20382 server_base.cc:1034] running on GCE node
W20250114 08:58:23.689908 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 08:58:23.692334 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 08:58:23.693105 20382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 08:58:23.693390 20382 hybrid_clock.cc:648] HybridClock initialized: now 1736845103693370 us; error 0 us; skew 500 ppm
I20250114 08:58:23.694211 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 08:58:23.713299 20382 webserver.cc:458] Webserver started at http://127.19.231.132:33951/ using document root <none> and password file <none>
I20250114 08:58:23.714082 20382 fs_manager.cc:362] Metadata directory not provided
I20250114 08:58:23.714388 20382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 08:58:23.714780 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 08:58:23.716703 20382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-3-root/instance:
uuid: "57f9cc1dd3c64647848a33a65b08c893"
format_stamp: "Formatted at 2025-01-14 08:58:23 on dist-test-slave-4wc2"
I20250114 08:58:23.723559 20382 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250114 08:58:23.735183 22236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:23.736222 20382 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.006s
I20250114 08:58:23.736653 20382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-3-root
uuid: "57f9cc1dd3c64647848a33a65b08c893"
format_stamp: "Formatted at 2025-01-14 08:58:23 on dist-test-slave-4wc2"
I20250114 08:58:23.737054 20382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 08:58:23.787354 20382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 08:58:23.789345 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 08:58:23.791462 20382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 08:58:23.795236 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 08:58:23.795585 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:23.795922 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 08:58:23.796154 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 08:58:23.972651 20382 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.132:35045
I20250114 08:58:23.972414 22299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.231.132:35045 every 8 connection(s)
I20250114 08:58:23.984658 20382 auto_leader_rebalancer-test.cc:413] add a tserver: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:23.999030 22300 heartbeater.cc:346] Connected to a master server at 127.19.231.190:44737
I20250114 08:58:23.999440 22300 heartbeater.cc:463] Registering TS with master...
I20250114 08:58:24.000197 22300 heartbeater.cc:510] Master 127.19.231.190:44737 requested a full tablet report, sending...
I20250114 08:58:24.002388 21450 ts_manager.cc:194] Registered new tserver with Master: 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:24.004349 21450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36438
I20250114 08:58:25.007028 22300 heartbeater.cc:502] Master 127.19.231.190:44737 was elected leader, sending a full tablet report...
I20250114 08:58:25.065505 21697 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.073493 21630 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:25.075373 22176 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:25.075892 21553 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:25.078624 22176 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:25.088151 22176 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.093319 21630 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
W20250114 08:58:25.090469 21667 consensus_peers.cc:487] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6846f3a6d80748369341fe169ebc427e. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:25.113340 21450 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.106727 21553 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.177860 21625 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.194090 21551 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:25.195461 21697 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:25.196977 22052 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:25.203744 22052 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 08:58:25.207267 21593 consensus_peers.cc:487] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e4913bb74ad4fab98dd737eb0bd5965. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:25.224256 22052 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.225769 21553 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.242197 21625 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.238515 21697 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.252142 21453 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.287532 21553 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:25.290846 22325 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 08:58:25.294602 21593 consensus_peers.cc:487] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 426decde999b4722a874569acae5664f. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:25.298403 21697 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:25.300719 22325 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:25.305325 22304 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.323019 21553 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.324498 21452 catalog_manager.cc:5526] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:25.372661 21551 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.351421 21697 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.385207 21704 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:25.385838 21625 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:25.387182 22136 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:25.388324 22238 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:25.415184 22136 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.417207 21699 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.447314 21630 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:25.467871 21452 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250114 08:58:25.478309 21518 consensus_peers.cc:487] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7929f46385a41b0acb9e10a87605e22. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:25.582356 22336 ts_tablet_manager.cc:927] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:25.585830 22336 tablet_copy_client.cc:323] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:25.604558 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet 6e4913bb74ad4fab98dd737eb0bd5965 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:25.605274 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet 6e4913bb74ad4fab98dd737eb0bd5965 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-6e4913bb74ad4fab98dd737eb0bd5965
I20250114 08:58:25.617496 21640 tablet_copy_source_session.cc:215] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:25.622407 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4913bb74ad4fab98dd737eb0bd5965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:25.642087 22336 tablet_copy_client.cc:806] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:25.642840 22336 tablet_copy_client.cc:670] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:25.648183 22336 tablet_copy_client.cc:538] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:25.653038 22339 ts_tablet_manager.cc:927] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:25.658082 22336 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:25.671267 22339 tablet_copy_client.cc:323] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.131:37485
I20250114 08:58:25.709779 22336 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:25.711422 22336 tablet_bootstrap.cc:492] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:25.712270 22336 ts_tablet_manager.cc:1397] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.054s user 0.022s sys 0.007s
I20250114 08:58:25.716614 22336 raft_consensus.cc:357] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.717731 21714 tablet_copy_service.cc:140] P cec0ef3bbb654988bf2123d8eef7a0a7: Received BeginTabletCopySession request for tablet 6846f3a6d80748369341fe169ebc427e from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:45060)
I20250114 08:58:25.718365 21714 tablet_copy_service.cc:161] P cec0ef3bbb654988bf2123d8eef7a0a7: Beginning new tablet copy session on tablet 6846f3a6d80748369341fe169ebc427e from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:45060: session id = 57f9cc1dd3c64647848a33a65b08c893-6846f3a6d80748369341fe169ebc427e
I20250114 08:58:25.717792 22336 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:25.721503 22336 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.725728 21714 tablet_copy_source_session.cc:215] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:25.729909 22336 ts_tablet_manager.cc:1428] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20250114 08:58:25.734086 22339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846f3a6d80748369341fe169ebc427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:25.736533 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-6e4913bb74ad4fab98dd737eb0bd5965 received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:25.737118 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-6e4913bb74ad4fab98dd737eb0bd5965 on tablet 6e4913bb74ad4fab98dd737eb0bd5965 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:25.755462 22339 tablet_copy_client.cc:806] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:25.756101 22339 tablet_copy_client.cc:670] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:25.761330 22339 tablet_copy_client.cc:538] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:25.768805 22339 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:25.786557 22336 ts_tablet_manager.cc:927] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:25.791081 22339 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:25.792098 22339 tablet_bootstrap.cc:492] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:25.792855 22339 ts_tablet_manager.cc:1397] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.009s
I20250114 08:58:25.796736 22339 raft_consensus.cc:357] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.797689 22339 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:25.798391 22339 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.801421 22339 ts_tablet_manager.cc:1428] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250114 08:58:25.804668 21714 tablet_copy_service.cc:342] P cec0ef3bbb654988bf2123d8eef7a0a7: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-6846f3a6d80748369341fe169ebc427e received from {username='slave'} at 127.0.0.1:45060
I20250114 08:58:25.805104 21714 tablet_copy_service.cc:434] P cec0ef3bbb654988bf2123d8eef7a0a7: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-6846f3a6d80748369341fe169ebc427e on tablet 6846f3a6d80748369341fe169ebc427e with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:25.806574 22336 tablet_copy_client.cc:323] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.129:45539
I20250114 08:58:25.824340 21565 tablet_copy_service.cc:140] P 28a83b7a7d9d4aa899b425086aa45a8e: Received BeginTabletCopySession request for tablet b7929f46385a41b0acb9e10a87605e22 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:50144)
I20250114 08:58:25.824991 21565 tablet_copy_service.cc:161] P 28a83b7a7d9d4aa899b425086aa45a8e: Beginning new tablet copy session on tablet b7929f46385a41b0acb9e10a87605e22 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:50144: session id = 57f9cc1dd3c64647848a33a65b08c893-b7929f46385a41b0acb9e10a87605e22
I20250114 08:58:25.835779 21565 tablet_copy_source_session.cc:215] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:25.840265 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7929f46385a41b0acb9e10a87605e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:25.846998 22339 ts_tablet_manager.cc:927] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:25.862377 22339 tablet_copy_client.cc:323] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:25.865257 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet 426decde999b4722a874569acae5664f from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:25.865922 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet 426decde999b4722a874569acae5664f from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-426decde999b4722a874569acae5664f
I20250114 08:58:25.876504 21640 tablet_copy_source_session.cc:215] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:25.880326 22339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426decde999b4722a874569acae5664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:25.896477 22336 tablet_copy_client.cc:806] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:25.897133 22336 tablet_copy_client.cc:670] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:25.908044 22336 tablet_copy_client.cc:538] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:25.917865 22336 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:25.932009 22339 tablet_copy_client.cc:806] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:25.932860 22339 tablet_copy_client.cc:670] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:25.936690 22339 tablet_copy_client.cc:538] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:25.947001 22339 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:25.973520 22336 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:25.974815 22336 tablet_bootstrap.cc:492] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:25.975719 22336 ts_tablet_manager.cc:1397] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.058s user 0.028s sys 0.011s
I20250114 08:58:25.984108 22336 raft_consensus.cc:357] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.985143 22336 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:25.985844 22336 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:25.988068 22336 ts_tablet_manager.cc:1428] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20250114 08:58:25.990716 21565 tablet_copy_service.cc:342] P 28a83b7a7d9d4aa899b425086aa45a8e: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-b7929f46385a41b0acb9e10a87605e22 received from {username='slave'} at 127.0.0.1:50144
I20250114 08:58:25.991181 21565 tablet_copy_service.cc:434] P 28a83b7a7d9d4aa899b425086aa45a8e: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-b7929f46385a41b0acb9e10a87605e22 on tablet b7929f46385a41b0acb9e10a87605e22 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:25.995805 22339 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:25.996836 22339 tablet_bootstrap.cc:492] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:25.998085 22339 ts_tablet_manager.cc:1397] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.025s
I20250114 08:58:26.001124 22339 raft_consensus.cc:357] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:26.002131 22339 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:26.002866 22339 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:26.005607 22339 ts_tablet_manager.cc:1428] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 08:58:26.007354 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-426decde999b4722a874569acae5664f received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:26.007807 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-426decde999b4722a874569acae5664f on tablet 426decde999b4722a874569acae5664f with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:26.119150 22275 raft_consensus.cc:1212] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:26.162724 22275 raft_consensus.cc:1212] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:26.368603 22275 raft_consensus.cc:1212] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250114 08:58:26.377449 22275 raft_consensus.cc:1212] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:26.706682 22317 raft_consensus.cc:1059] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:26.711593 22317 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:26.735026 21551 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.737844 22275 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.739074 21628 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.740676 22350 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:26.742054 22217 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:26.744774 22317 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:26.751480 22349 raft_consensus.cc:1059] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:26.754669 22349 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:26.774103 22317 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.775683 22136 raft_consensus.cc:1059] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:26.778110 22136 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:26.784343 22274 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.785962 22349 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:26.788434 22274 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.790786 21551 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.794052 22349 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:26.810263 21699 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.817631 21553 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.822476 21453 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.830380 22052 raft_consensus.cc:1059] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:26.832592 22304 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:26.835732 22052 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:26.842344 21630 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.852193 22274 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.856228 21699 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:26.856925 22304 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:26.857844 22356 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:26.862022 21699 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.862371 22275 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:26.864003 22349 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:26.867799 21553 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:26.867043 22349 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.869235 22304 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:26.872077 21697 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.875725 22333 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:26.884996 21626 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:26.887512 22356 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:26.905211 21555 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.917702 22274 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.969449 21699 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.967443 22304 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.988195 22333 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.991796 21704 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.992969 21554 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:26.984081 21698 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:27.013415 21453 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.010381 21626 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.016585 22275 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.034484 22274 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:27.037518 22350 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:27.038604 21555 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:27.044081 22317 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:27.051736 22353 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.059425 21555 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.064546 22275 raft_consensus.cc:2949] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.073987 21453 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.094519 21440 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 6846f3a6d80748369341fe169ebc427e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:27.097503 22274 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.104873 21452 catalog_manager.cc:5526] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.112696 21626 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:27.117661 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 52306.4us
I20250114 08:58:27.131234 22275 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:27.133102 22349 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:27.135440 21554 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:27.137428 22304 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:27.142369 22052 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.145565 22274 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.165266 21451 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:27.171779 21555 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:27.179373 22274 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:58:27.182956 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 6e4913bb74ad4fab98dd737eb0bd5965 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:27.183456 21554 raft_consensus.cc:2949] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.185611 21450 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.187393 21618 tablet_service.cc:1514] Processing DeleteTablet for tablet 6846f3a6d80748369341fe169ebc427e with delete_type TABLET_DATA_TOMBSTONED (TS d7732324fb5f48d3b16a29d4bb2584ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57308
I20250114 08:58:27.188922 22333 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:27.201649 21694 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4913bb74ad4fab98dd737eb0bd5965 with delete_type TABLET_DATA_TOMBSTONED (TS cec0ef3bbb654988bf2123d8eef7a0a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46386
I20250114 08:58:27.201237 21625 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:27.205935 21697 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:58:27.207742 22356 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:27.211623 22275 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:27.213449 22377 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:27.213598 22305 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.227365 21697 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.227667 22377 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.238075 21438 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b7929f46385a41b0acb9e10a87605e22 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:27.242400 21698 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:27.245425 22273 raft_consensus.cc:2949] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.238049 22275 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.265926 22349 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:27.276964 21704 raft_consensus.cc:2949] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.315690 21450 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 4 to 5, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.327148 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 426decde999b4722a874569acae5664f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:27.342043 22379 tablet_replica.cc:331] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: stopping tablet replica
I20250114 08:58:27.354081 22379 raft_consensus.cc:2238] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:27.344933 22380 tablet_replica.cc:331] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: stopping tablet replica
I20250114 08:58:27.354761 22379 raft_consensus.cc:2267] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:27.350651 21618 tablet_service.cc:1514] Processing DeleteTablet for tablet b7929f46385a41b0acb9e10a87605e22 with delete_type TABLET_DATA_TOMBSTONED (TS d7732324fb5f48d3b16a29d4bb2584ed not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57308
I20250114 08:58:27.355687 22380 raft_consensus.cc:2238] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:27.356514 22380 raft_consensus.cc:2267] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:27.358086 22379 ts_tablet_manager.cc:1905] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:27.360736 22380 ts_tablet_manager.cc:1905] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:27.373667 22379 ts_tablet_manager.cc:1918] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:27.374330 22379 log.cc:1198] T 6846f3a6d80748369341fe169ebc427e P d7732324fb5f48d3b16a29d4bb2584ed: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root/wals/6846f3a6d80748369341fe169ebc427e
I20250114 08:58:27.376291 21439 catalog_manager.cc:4872] TS d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): tablet 6846f3a6d80748369341fe169ebc427e (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:27.376297 22379 tablet_replica.cc:331] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: stopping tablet replica
I20250114 08:58:27.378106 22379 raft_consensus.cc:2238] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:27.378921 22379 raft_consensus.cc:2267] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:27.383611 22379 ts_tablet_manager.cc:1905] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:27.384408 22380 ts_tablet_manager.cc:1918] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:27.384950 22380 log.cc:1198] T 6e4913bb74ad4fab98dd737eb0bd5965 P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root/wals/6e4913bb74ad4fab98dd737eb0bd5965
I20250114 08:58:27.413762 22379 ts_tablet_manager.cc:1918] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20250114 08:58:27.414302 22379 log.cc:1198] T b7929f46385a41b0acb9e10a87605e22 P d7732324fb5f48d3b16a29d4bb2584ed: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root/wals/b7929f46385a41b0acb9e10a87605e22
I20250114 08:58:27.464605 21453 catalog_manager.cc:5526] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:27.386775 21440 catalog_manager.cc:4872] TS cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): tablet 6e4913bb74ad4fab98dd737eb0bd5965 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
W20250114 08:58:27.466477 21440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 79732.5us
I20250114 08:58:28.471550 22317 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.472221 22317 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.474653 22317 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.494019 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 534) took 1080 ms. Trace:
I20250114 08:58:28.494408 21704 rpcz_store.cc:276] 0114 08:58:27.473374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:28.493144 (+1019770us) service_pool.cc:225] Handling call
0114 08:58:28.493522 (+ 378us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:28.493704 (+ 182us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:28.493731 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:28.493769 (+ 38us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:28.493795 (+ 26us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:28.493831 (+ 36us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:28.493939 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:28.516805 22388 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.517544 22388 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.519860 22388 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.520349 22388 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.520864 22388 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.523313 22388 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.524032 22388 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.524540 22388 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.525395 22387 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.525879 22387 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.526194 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 535) took 1052 ms. Trace:
I20250114 08:58:28.526773 22388 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.527611 22387 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.526597 21697 rpcz_store.cc:276] 0114 08:58:27.473829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:28.525423 (+1051594us) service_pool.cc:225] Handling call
0114 08:58:28.525721 (+ 298us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:28.525883 (+ 162us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:28.525909 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:28.525946 (+ 37us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:28.525975 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:28.526025 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:28.526141 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:28.532678 22393 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.532761 22388 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.533797 22392 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.533890 22388 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.534515 22392 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.535065 22389 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.535739 22389 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.536801 22392 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.537711 22388 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.544799 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.545447 22390 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.546010 22390 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.546275 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.547536 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.547857 22390 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.550019 22356 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.550607 22356 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.551187 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.553093 22356 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.532881 22391 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.555804 22390 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.555630 22391 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.556340 22390 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.557054 22052 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.557525 22052 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.557842 22391 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.533469 22393 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.542024 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.558231 22390 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.559253 22052 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.560355 22393 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.560411 22389 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.570430 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.572978 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.576021 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.583508 22391 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.585018 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.584579 22391 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.593516 22391 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.595500 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.597767 22373 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.598575 22373 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.601481 22373 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.606387 22375 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.606899 22375 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.607568 22217 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.608055 22217 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.608861 22391 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.609593 22374 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.610109 22374 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.612018 22389 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.612545 22389 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.533416 21439 catalog_manager.cc:4872] TS d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): tablet b7929f46385a41b0acb9e10a87605e22 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
W20250114 08:58:28.614225 21439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 80832.1us
I20250114 08:58:28.615128 22391 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.616030 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 543) took 1074 ms. Trace:
I20250114 08:58:28.616775 22395 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.617555 22395 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.621764 21452 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55888 (request call id 149) took 1230 ms. Trace:
I20250114 08:58:28.622216 21452 rpcz_store.cc:276] 0114 08:58:27.391801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:27.391943 (+ 142us) service_pool.cc:225] Handling call
0114 08:58:28.621663 (+1229720us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69971,"reported_tablets":1}
I20250114 08:58:28.622954 22333 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.623482 22333 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.623921 22376 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.624411 22376 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.626377 22376 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.626675 22136 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.627156 22136 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.628743 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 537) took 1154 ms. Trace:
I20250114 08:58:28.629122 21699 rpcz_store.cc:276] 0114 08:58:27.474331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:28.619752 (+1145421us) service_pool.cc:225] Handling call
0114 08:58:28.620116 (+ 364us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:28.620335 (+ 219us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:28.620371 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:28.620424 (+ 53us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:28.628511 (+ 8087us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:28.628582 (+ 71us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:28.628685 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:28.616420 21701 rpcz_store.cc:276] 0114 08:58:27.541921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:28.615338 (+1073417us) service_pool.cc:225] Handling call
0114 08:58:28.615613 (+ 275us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:28.615760 (+ 147us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:28.615784 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:28.615818 (+ 34us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:28.615847 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:28.615892 (+ 45us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:28.615977 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:58:28.631768 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800219cc0 after lost signal to thread 20385
I20250114 08:58:28.637888 22317 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.638726 22375 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.638549 22317 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.643075 22391 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.678252 22391 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.678678 22391 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.643639 22395 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:28.643718 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080003b740 after lost signal to thread 21487
I20250114 08:58:28.643958 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.645009 22333 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.647266 22317 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.616971 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.680183 21700 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
W20250114 08:58:28.680496 21576 debug-util.cc:398] Leaking SignalData structure 0x7b08001ba3a0 after lost signal to thread 21579
I20250114 08:58:28.654644 22136 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.655452 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.646100 22374 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:28.681468 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800219c60 after lost signal to thread 21654
W20250114 08:58:28.681847 21576 debug-util.cc:398] Leaking SignalData structure 0x7b0800219c40 after lost signal to thread 21655
I20250114 08:58:28.681424 21596 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:28.682976 22396 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.664840 22389 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.665354 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.666980 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.669240 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.686439 22389 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.671409 22052 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.671970 22388 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.687077 22052 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.687142 22388 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.687863 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.672353 22393 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.689082 22052 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.689359 22388 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.689167 22393 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.689656 22052 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:28.690032 22052 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.690371 22052 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:28.691283 22393 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.673311 22356 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.672446 22392 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.674216 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.675357 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.692580 22356 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.692035 22392 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.694429 22356 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.694626 22392 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.694795 22389 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.676023 22217 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.661936 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:28.682966 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080031a6e0 after lost signal to thread 21728
I20250114 08:58:28.695713 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:28.696699 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080031a2c0 after lost signal to thread 22299
I20250114 08:58:28.696874 22389 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:28.697005 21576 debug-util.cc:398] Leaking SignalData structure 0x7b080031a280 after lost signal to thread 22300
I20250114 08:58:28.697068 22217 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.697705 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.697855 22217 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.700268 22217 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.670470 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.662904 22394 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.701872 22394 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.706156 22394 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.699903 22052 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.707401 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.707942 22052 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.700507 22356 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.708595 22356 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:28.711066 22356 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:28.712515 22394 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.713174 22394 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.712034 22052 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.715554 22052 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.717988 22402 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.718955 22052 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.719470 22400 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.718784 22402 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.719913 22400 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:28.721829 22402 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.717244 22394 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.683830 22395 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.723291 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.723834 22356 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.757972 22356 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.758308 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.759940 22356 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.759855 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.761538 22395 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.763718 22395 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.763900 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.727562 22400 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.728847 22375 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.764989 22375 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:28.767481 22375 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:28.731297 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.732327 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.732790 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.768575 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:28.736054 22136 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.769953 22136 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.770620 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.771698 22136 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.736613 22333 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:28.772370 22333 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.773901 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.774333 22333 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.774772 21623 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:28.740026 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.742055 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.743561 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.777156 21552 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:28.745998 22407 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.747678 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:28.748221 22396 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:28.778232 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.748600 22391 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.748800 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.751008 21453 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 20) took 1397 ms. Trace:
I20250114 08:58:28.751390 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.779872 22396 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.780105 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.751746 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.780784 21623 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:28.781813 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.752846 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.753499 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 150) took 1355 ms. Trace:
I20250114 08:58:28.784353 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.784538 21451 rpcz_store.cc:276] 0114 08:58:27.397673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:27.397827 (+ 154us) service_pool.cc:225] Handling call
0114 08:58:28.753438 (+1355611us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67369,"reported_tablets":1}
I20250114 08:58:28.753912 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.778873 22407 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:28.754101 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.772631 22395 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.740550 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.786967 22395 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:28.787868 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 95312.1us
I20250114 08:58:28.787808 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.789124 22395 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.789570 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.790047 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.790287 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.790992 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.791857 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.792781 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.794323 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:28.794790 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.795204 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.795277 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.794457 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.795678 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.795902 22239 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.796388 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.796515 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.797060 22400 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:28.797451 22400 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:28.797809 22400 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:28.799386 22391 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.799882 22391 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.800791 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.801223 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.801815 22396 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.802556 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.803668 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.803990 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.804761 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.801854 22391 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.805631 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.805822 22391 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.805861 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.807641 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.806406 22391 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.787285 22407 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:28.809317 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.799235 22395 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.779872 21453 rpcz_store.cc:276] 0114 08:58:27.353197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:27.353344 (+ 147us) service_pool.cc:225] Handling call
0114 08:58:28.750946 (+1397602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:27.526244 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:27.526585 (+ 341us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:27.526609 (+ 24us) write_op.cc:435] Acquired schema lock
0114 08:58:27.526619 (+ 10us) tablet.cc:659] Decoding operations
0114 08:58:27.526854 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:27.526890 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:27.526907 (+ 17us) tablet.cc:682] Acquiring locks for 1 operations
0114 08:58:27.527037 (+ 130us) tablet.cc:698] Row locks acquired
0114 08:58:27.527045 (+ 8us) write_op.cc:260] PREPARE: finished
0114 08:58:27.527135 (+ 90us) write_op.cc:270] Start()
0114 08:58:27.527215 (+ 80us) write_op.cc:276] Timestamp: P: 1736845107527107 usec, L: 0
0114 08:58:27.527225 (+ 10us) op_driver.cc:348] REPLICATION: starting
0114 08:58:27.527739 (+ 514us) log.cc:844] Serialized 538 byte log entry
0114 08:58:28.670616 (+1142877us) op_driver.cc:464] REPLICATION: finished
0114 08:58:28.679703 (+ 9087us) write_op.cc:301] APPLY: starting
0114 08:58:28.679785 (+ 82us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:28.683613 (+ 3828us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:28.683782 (+ 169us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:28.684023 (+ 241us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:28.684213 (+ 190us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 08:58:28.684296 (+ 83us) write_op.cc:312] APPLY: finished
0114 08:58:28.684600 (+ 304us) log.cc:844] Serialized 22 byte log entry
0114 08:58:28.748944 (+ 64344us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:28.749003 (+ 59us) write_op.cc:454] Released schema lock
0114 08:58:28.749032 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:28.749349 (+ 317us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:28.749382 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":67267,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":8805,"apply.run_cpu_time_us":1778,"apply.run_wall_time_us":69632,"num_ops":1,"prepare.queue_time_us":58542,"prepare.run_cpu_time_us":1691,"prepare.run_wall_time_us":1691,"replication_time_us":1143241,"thread_start_us":10979,"threads_started":1}]]}
I20250114 08:58:28.810372 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.810829 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.811977 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.810070 22395 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.813621 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.813891 22400 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:28.815177 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.815383 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.816812 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:28.817116 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:28.818272 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.818500 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.819375 22396 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.819684 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.820664 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.821709 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.822012 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.821745 22391 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.821298 22395 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:28.823652 22400 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.824556 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.825872 21521 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.826581 22375 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.827620 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.828390 21520 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.829339 22356 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.829456 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.831287 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.832068 22356 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.834026 21520 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.834874 22356 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.835805 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:28.836789 21520 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.837756 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.838861 22390 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:28.839275 22390 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.838286 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:28.839416 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:28.839663 22390 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:28.838232 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.840167 22350 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:28.838495 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.840584 22350 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:28.841071 22350 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:28.842020 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.843396 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.845741 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.846807 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.846706 22390 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.847549 21594 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.848006 22350 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:28.849505 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:28.850078 22395 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.851573 22390 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.852679 22356 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.853118 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.853593 21594 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.855365 21594 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.856103 22391 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:28.856511 21594 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.856829 22391 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.857429 22406 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.857723 21594 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.858500 22406 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.858904 21594 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.859674 22406 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.860087 21594 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.860690 22356 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.860957 22395 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.861436 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.862027 22375 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.862280 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.862195 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.862988 22375 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:28.863086 22395 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.854012 22350 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:28.864553 21594 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.864868 21520 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.865542 22375 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.865542 22406 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.867977 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.868057 21594 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.868793 22356 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.872121 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.868902 22406 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.873209 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.874892 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.875829 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:28.876927 21668 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.878388 21668 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.879976 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet 426decde999b4722a874569acae5664f with delete_type TABLET_DATA_TOMBSTONED (TS 28a83b7a7d9d4aa899b425086aa45a8e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33868
I20250114 08:58:28.881155 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.881251 22402 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:28.882283 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:28.882556 22402 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.882974 21668 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.883445 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:28.883819 22402 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.885792 22333 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.888816 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.889721 22375 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.890044 21668 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.891086 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.891566 22387 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.891882 22375 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.898401 22408 tablet_replica.cc:331] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: stopping tablet replica
I20250114 08:58:28.876571 21594 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.899484 22408 raft_consensus.cc:2238] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:28.899736 21668 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.900565 22406 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.900856 22408 raft_consensus.cc:2267] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:28.901409 22390 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.902108 21668 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.902679 21594 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.903297 22390 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:28.903805 22408 ts_tablet_manager.cc:1905] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:28.905081 22406 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.913282 21594 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.914083 22395 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.914485 21594 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.915292 22395 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.915688 21594 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.916453 22395 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.916882 21594 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.917706 22406 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.918815 21594 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.919591 22406 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:28.920877 21594 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.921644 22406 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.923604 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.923796 22408 ts_tablet_manager.cc:1918] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:28.924321 22408 log.cc:1198] T 426decde999b4722a874569acae5664f P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/wals/426decde999b4722a874569acae5664f
I20250114 08:58:28.924387 22406 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.924795 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.925648 22406 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.926380 21438 catalog_manager.cc:4872] TS 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): tablet 426decde999b4722a874569acae5664f (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:28.927191 21594 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.927909 22406 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.928493 21669 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.929195 22350 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.929877 21669 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.930663 22350 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.934319 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.937600 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.940351 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.940779 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.942029 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:28.944530 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.945492 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:28.942777 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:28.946889 22395 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.948179 21594 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.949028 22395 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:28.950762 21594 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.951550 22406 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.951908 21594 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.952723 22406 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.954295 21594 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:28.955078 22395 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.957494 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.958240 22390 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.959472 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:28.960259 22390 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:28.993016 22275 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:28.994262 22395 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:28.995687 22406 raft_consensus.cc:3049] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:28.996424 22406 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:28.997397 22406 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:28.998961 22406 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:29.023015 21700 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:29.025125 22356 consensus_queue.cc:1046] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:29.026217 22375 raft_consensus.cc:3049] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:29.026607 22375 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:29.027266 22375 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:29.028255 22375 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:29.033874 22390 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:29.034585 22390 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:29.036077 22390 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:29.036964 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:29.037156 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:29.038620 21669 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:29.039395 22390 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:29.047811 22350 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:29.048516 22350 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:29.050750 22350 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:29.052706 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:29.055455 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:29.056860 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:29.057483 22350 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:29.078121 22350 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:29.078563 22350 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:29.080088 22350 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:29.081013 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:29.081141 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:29.083758 21668 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:29.085778 22350 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:29.127816 21623 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:29.132124 22356 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:29.133213 22333 raft_consensus.cc:3049] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:29.133563 22333 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:29.134261 22333 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:29.135592 22333 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:29.292590 21696 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:29.294314 22333 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:29.295307 22356 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:29.295686 22356 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:29.296484 22356 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:29.297724 22356 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:29.566830 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1060 after lost signal to thread 20385
W20250114 08:58:29.567961 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1740 after lost signal to thread 21487
W20250114 08:58:29.569252 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002b7c80 after lost signal to thread 21579
W20250114 08:58:29.570389 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002f9720 after lost signal to thread 21654
W20250114 08:58:29.570690 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002f96e0 after lost signal to thread 21655
W20250114 08:58:29.571794 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002cbd40 after lost signal to thread 21728
W20250114 08:58:29.572099 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002bb860 after lost signal to thread 21729
W20250114 08:58:29.573266 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002cbf60 after lost signal to thread 22299
W20250114 08:58:29.573622 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002cc820 after lost signal to thread 22300
W20250114 08:58:29.715957 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 74440us
W20250114 08:58:29.723753 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 81559.9us
I20250114 08:58:29.808387 21696 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
W20250114 08:58:29.874538 21667 consensus_peers.cc:487] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc5506492f514407a5097b5b2e8c7042. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:29.874720 21624 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:29.876698 22350 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:29.876911 21549 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:29.880402 22350 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:29.896040 22387 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:29.917536 21549 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:29.934676 21453 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:29.959393 21625 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
W20250114 08:58:29.988323 21593 consensus_peers.cc:487] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f21ba5689065467b967a5802582959dd. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:29.993520 21696 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:29.995200 22391 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:29.998272 21624 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.007359 21549 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:30.010077 22406 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:30.022145 22406 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.031219 21696 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.045929 21549 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.067229 21696 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:30.081064 21624 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250114 08:58:30.081909 21667 consensus_peers.cc:487] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e77833a006f478e86764142c505ab6b. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:30.082652 22390 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:30.087433 21554 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:30.096746 22350 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:30.146417 21453 catalog_manager.cc:5526] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.161974 22390 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.165975 21549 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.179023 21625 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.186419 21451 catalog_manager.cc:5526] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.213490 21549 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:30.234189 21700 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:30.235848 22410 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:30.251505 21624 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250114 08:58:30.252118 21518 consensus_peers.cc:487] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d19fa6dec61d44618fee387c9b1f2994. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:30.253191 22409 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:30.265498 22411 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.274824 22356 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:30.275468 22356 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.279131 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:30.279928 21700 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:30.283895 22356 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.284742 21521 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:30.285545 22333 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:30.285872 22333 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:30.286165 22333 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.295513 21696 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.303900 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:30.309865 21625 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:30.312261 22413 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:30.312997 22413 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:30.316536 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:30.316737 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:30.317255 21549 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:30.318336 22413 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:30.319000 21594 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:30.317342 22275 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:30.319787 22413 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:30.320153 22413 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:30.320518 22413 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.338663 21453 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:30.346890 22413 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:30.353260 22413 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:30.355110 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:30.356063 22275 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.362339 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:30.363109 21549 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.373056 22406 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:30.373632 22406 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.375808 22406 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.377473 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:30.379695 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:30.380859 21703 raft_consensus.cc:2388] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:30.382408 22333 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.383517 21554 raft_consensus.cc:2363] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:30.386137 21594 leader_election.cc:400] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:30.386488 21594 leader_election.cc:403] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:58:30.387166 22413 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.394484 22413 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:30.407828 22275 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:30.409678 21593 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:30.411150 22406 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:30.411804 22406 raft_consensus.cc:695] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:30.412887 22406 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:30.425670 22432 ts_tablet_manager.cc:927] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.418313 21549 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:30.444558 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:30.445345 21696 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.444690 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:30.451856 22333 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.460610 21696 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:30.460891 22432 tablet_copy_client.cc:323] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.131:37485
I20250114 08:58:30.462872 21714 tablet_copy_service.cc:140] P cec0ef3bbb654988bf2123d8eef7a0a7: Received BeginTabletCopySession request for tablet cc5506492f514407a5097b5b2e8c7042 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:45060)
I20250114 08:58:30.463508 21714 tablet_copy_service.cc:161] P cec0ef3bbb654988bf2123d8eef7a0a7: Beginning new tablet copy session on tablet cc5506492f514407a5097b5b2e8c7042 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:45060: session id = 57f9cc1dd3c64647848a33a65b08c893-cc5506492f514407a5097b5b2e8c7042
I20250114 08:58:30.465198 21521 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:30.466097 22333 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:30.466404 21624 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:30.466573 22333 raft_consensus.cc:695] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:30.468945 22333 consensus_queue.cc:237] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.472117 21714 tablet_copy_source_session.cc:215] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:30.484673 22432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5506492f514407a5097b5b2e8c7042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:30.542695 22356 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:30.544936 22356 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.549937 21451 catalog_manager.cc:5526] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:30.570794 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:30.572325 21623 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:30.579219 22356 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.580592 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:30.581611 21703 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:30.575114 21453 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:30.589309 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:30.597642 22432 tablet_copy_client.cc:806] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:30.598819 22410 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:30.600301 22410 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:30.600732 22410 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.601670 22432 tablet_copy_client.cc:670] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:30.625782 22410 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.646106 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:30.646885 21703 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.659770 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:30.660722 21624 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.661168 22410 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.679734 22432 tablet_copy_client.cc:538] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:30.682111 21703 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:30.683565 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:30.684514 22356 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:30.685070 22356 raft_consensus.cc:695] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:30.686236 22356 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.695674 21624 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:30.708969 22432 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:30.733512 22439 ts_tablet_manager.cc:927] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.733578 22438 ts_tablet_manager.cc:927] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:30.743703 22437 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:30.744491 22437 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.748620 22437 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:30.755044 22413 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:30.754848 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:30.755988 22413 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.757110 21623 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:30.759690 22413 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.762647 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:30.763550 21549 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:30.765893 21453 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:30.771737 21668 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:30.774298 22437 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:30.775012 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:30.781138 22437 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:30.781667 22437 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.792770 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:30.793507 21549 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:30.796006 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:30.796926 22440 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:30.797322 22440 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:30.797783 22440 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:30.817456 22445 ts_tablet_manager.cc:927] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:30.827833 22439 tablet_copy_client.cc:323] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.131:37485
I20250114 08:58:30.841158 22440 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.843464 22440 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:30.845027 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:30.857261 21551 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:30.858646 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:30.864859 21714 tablet_copy_service.cc:140] P cec0ef3bbb654988bf2123d8eef7a0a7: Received BeginTabletCopySession request for tablet 2e77833a006f478e86764142c505ab6b from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:45060)
I20250114 08:58:30.865581 21714 tablet_copy_service.cc:161] P cec0ef3bbb654988bf2123d8eef7a0a7: Beginning new tablet copy session on tablet 2e77833a006f478e86764142c505ab6b from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:45060: session id = 57f9cc1dd3c64647848a33a65b08c893-2e77833a006f478e86764142c505ab6b
I20250114 08:58:30.866228 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:30.852612 22437 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.868412 21703 raft_consensus.cc:2363] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:30.871899 22437 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:30.875123 22438 tablet_copy_client.cc:323] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:30.876405 21596 leader_election.cc:400] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:30.878454 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet f21ba5689065467b967a5802582959dd from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:30.879062 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet f21ba5689065467b967a5802582959dd from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-f21ba5689065467b967a5802582959dd
I20250114 08:58:30.884958 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:30.875169 22445 tablet_copy_client.cc:323] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.129:45539
I20250114 08:58:30.887777 21565 tablet_copy_service.cc:140] P 28a83b7a7d9d4aa899b425086aa45a8e: Received BeginTabletCopySession request for tablet d19fa6dec61d44618fee387c9b1f2994 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:50144)
I20250114 08:58:30.888142 21554 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:58:30.888386 21565 tablet_copy_service.cc:161] P 28a83b7a7d9d4aa899b425086aa45a8e: Beginning new tablet copy session on tablet d19fa6dec61d44618fee387c9b1f2994 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:50144: session id = 57f9cc1dd3c64647848a33a65b08c893-d19fa6dec61d44618fee387c9b1f2994
I20250114 08:58:30.891076 21640 tablet_copy_source_session.cc:215] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:30.872800 22413 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:30.893363 22413 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:30.893479 21700 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:30.894408 22413 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.895864 22356 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:30.898694 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:30.899348 21551 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:30.904341 22440 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:30.905645 21551 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:30.906275 21565 tablet_copy_source_session.cc:215] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:30.907294 22273 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:58:30.908820 22443 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:30.912762 21668 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:30.914297 22350 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:30.914747 22350 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:30.924345 22438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21ba5689065467b967a5802582959dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:30.952586 22350 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:30.953385 21714 tablet_copy_source_session.cc:215] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:30.976411 22445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19fa6dec61d44618fee387c9b1f2994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:30.987754 22439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e77833a006f478e86764142c505ab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:31.073254 21622 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 08:58:31.088634 22451 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20250114 08:58:31.117964 22438 tablet_copy_client.cc:806] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:31.118669 22438 tablet_copy_client.cc:670] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
W20250114 08:58:31.169152 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 126812us
I20250114 08:58:31.174060 22438 tablet_copy_client.cc:538] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:31.182297 22432 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:31.194270 22438 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:31.198740 22432 tablet_bootstrap.cc:492] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:31.199709 22432 ts_tablet_manager.cc:1397] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.491s user 0.021s sys 0.014s
I20250114 08:58:31.203087 22458 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:31.206941 22439 tablet_copy_client.cc:806] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:31.223881 22439 tablet_copy_client.cc:670] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:31.262507 21625 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 08:58:31.268770 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 46268us
I20250114 08:58:31.203092 22432 raft_consensus.cc:357] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:31.270867 22432 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:31.271596 22432 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:31.274721 22432 ts_tablet_manager.cc:1428] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.075s user 0.000s sys 0.007s
I20250114 08:58:31.278545 21714 tablet_copy_service.cc:342] P cec0ef3bbb654988bf2123d8eef7a0a7: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-cc5506492f514407a5097b5b2e8c7042 received from {username='slave'} at 127.0.0.1:45060
I20250114 08:58:31.279196 21714 tablet_copy_service.cc:434] P cec0ef3bbb654988bf2123d8eef7a0a7: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-cc5506492f514407a5097b5b2e8c7042 on tablet cc5506492f514407a5097b5b2e8c7042 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:31.300320 22410 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20250114 08:58:31.327464 22439 tablet_copy_client.cc:538] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:31.330715 21701 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:31.342692 22445 tablet_copy_client.cc:806] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:31.343502 22445 tablet_copy_client.cc:670] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:31.389011 22465 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250114 08:58:31.405092 22439 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:31.443220 22438 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:31.444114 22438 tablet_bootstrap.cc:492] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:31.444836 22438 ts_tablet_manager.cc:1397] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.251s user 0.019s sys 0.012s
I20250114 08:58:31.672930 22439 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:31.673955 22439 tablet_bootstrap.cc:492] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:31.679571 22439 ts_tablet_manager.cc:1397] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.275s user 0.015s sys 0.024s
W20250114 08:58:31.703491 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 326053us
I20250114 08:58:31.728351 21554 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:31.750371 22457 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:31.751386 22438 raft_consensus.cc:357] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:31.778273 22275 raft_consensus.cc:1212] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:31.778412 22438 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:31.779412 22438 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:31.751446 22442 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:31.782461 21701 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:31.782511 22442 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:31.783620 22442 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:31.784987 22442 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:31.757113 22439 raft_consensus.cc:357] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:31.760425 21554 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:31.788719 22439 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:31.789507 22439 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:31.794540 21452 catalog_manager.cc:5526] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 2, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 2 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
W20250114 08:58:31.798053 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 59181.2us
I20250114 08:58:31.809731 22445 tablet_copy_client.cc:538] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:31.883642 22439 ts_tablet_manager.cc:1428] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.192s user 0.005s sys 0.009s
I20250114 08:58:31.925352 22445 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:31.946014 22438 ts_tablet_manager.cc:1428] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.501s user 0.010s sys 0.003s
W20250114 08:58:31.946766 22438 ts_tablet_manager.cc:1460] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Tablet startup took 501ms
I20250114 08:58:31.951539 22473 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250114 08:58:31.958276 21630 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 08:58:31.958735 21630 pending_rounds.cc:77] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 1
I20250114 08:58:31.959015 21630 pending_rounds.cc:99] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:58:31.959283 21630 raft_consensus.cc:2883] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250114 08:58:31.947271 22438 ts_tablet_manager.cc:1462] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Trace:
0114 08:58:30.613148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:30.615990 (+ 2842us) service_pool.cc:225] Handling call
0114 08:58:30.741356 (+125366us) ts_tablet_manager.cc:928] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
0114 08:58:30.902771 (+161415us) tablet_copy_client.cc:368] Tablet copy session begun
0114 08:58:31.030026 (+127255us) tablet_metadata.cc:641] Metadata flushed
0114 08:58:31.030381 (+ 355us) tablet_copy_client.cc:497] Wrote new tablet metadata
0114 08:58:31.116961 (+ 86580us) tablet_copy_client.cc:418] Wrote new consensus metadata
0114 08:58:31.117316 (+ 355us) tablet_replica.cc:206] Creating consensus instance
0114 08:58:31.117662 (+ 346us) ts_tablet_manager.cc:983] Replica registered.
0114 08:58:31.117731 (+ 69us) inbound_call.cc:173] Queueing success response
0114 08:58:31.193867 (+ 76136us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0114 08:58:31.445171 (+251304us) ts_tablet_manager.cc:1429] Starting tablet replica
0114 08:58:31.445810 (+ 639us) tablet_replica.cc:260] Starting instrumentation
0114 08:58:31.446446 (+ 636us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":15230,"mutex_wait_us":54,"raft.queue_time_us":19249,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":542,"spinlock_wait_cycles":37632,"tablet-copy.queue_time_us":115265,"tablet-download-pool.queue_time_us":132,"tablet-download-pool.run_cpu_time_us":1169,"tablet-download-pool.run_wall_time_us":50892,"thread_start_us":126486,"threads_started":2}
I20250114 08:58:31.982250 22275 raft_consensus.cc:1212] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:31.986313 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-f21ba5689065467b967a5802582959dd received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:31.986793 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-f21ba5689065467b967a5802582959dd on tablet f21ba5689065467b967a5802582959dd with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:32.007156 21714 tablet_copy_service.cc:342] P cec0ef3bbb654988bf2123d8eef7a0a7: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-2e77833a006f478e86764142c505ab6b received from {username='slave'} at 127.0.0.1:45060
I20250114 08:58:32.007638 21714 tablet_copy_service.cc:434] P cec0ef3bbb654988bf2123d8eef7a0a7: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-2e77833a006f478e86764142c505ab6b on tablet 2e77833a006f478e86764142c505ab6b with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:32.049526 22350 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:32.099618 21453 catalog_manager.cc:5526] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 2 to 3, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 3 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:32.143003 22275 raft_consensus.cc:1212] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:32.155423 22445 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:32.159130 22445 tablet_bootstrap.cc:492] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:32.159920 22445 ts_tablet_manager.cc:1397] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.235s user 0.024s sys 0.008s
I20250114 08:58:32.175005 22445 raft_consensus.cc:357] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:32.175928 22445 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:32.176611 22445 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:32.184168 22445 ts_tablet_manager.cc:1428] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.024s user 0.002s sys 0.003s
I20250114 08:58:32.189810 21565 tablet_copy_service.cc:342] P 28a83b7a7d9d4aa899b425086aa45a8e: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-d19fa6dec61d44618fee387c9b1f2994 received from {username='slave'} at 127.0.0.1:50144
I20250114 08:58:32.190317 21565 tablet_copy_service.cc:434] P 28a83b7a7d9d4aa899b425086aa45a8e: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-d19fa6dec61d44618fee387c9b1f2994 on tablet d19fa6dec61d44618fee387c9b1f2994 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:32.317523 22387 raft_consensus.cc:1059] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:32.323495 22387 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:32.349973 22275 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.352552 22473 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:32.356321 21627 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.362951 22387 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:32.363376 21552 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.372543 22387 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:32.393930 22437 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.396037 21627 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.429419 22390 raft_consensus.cc:1059] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:32.432211 22390 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:32.431994 21450 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.445078 22275 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:58:32.449833 21484 debug-util.cc:398] Leaking SignalData structure 0x7b080003abe0 after lost signal to thread 20385
W20250114 08:58:32.450940 21484 debug-util.cc:398] Leaking SignalData structure 0x7b0800039e20 after lost signal to thread 21487
W20250114 08:58:32.452153 21484 debug-util.cc:398] Leaking SignalData structure 0x7b0800310c40 after lost signal to thread 21579
W20250114 08:58:32.453254 21484 debug-util.cc:398] Leaking SignalData structure 0x7b080032ff20 after lost signal to thread 21654
W20250114 08:58:32.454489 21484 debug-util.cc:398] Leaking SignalData structure 0x7b0800331aa0 after lost signal to thread 21728
W20250114 08:58:32.455607 21484 debug-util.cc:398] Leaking SignalData structure 0x7b08003179c0 after lost signal to thread 22299
I20250114 08:58:32.455461 21552 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.471941 21630 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.473345 21549 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.473461 22437 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:32.474929 22273 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.476631 22437 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:32.479872 22390 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:32.529408 22390 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.533031 21630 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.539885 21698 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:32.560564 21630 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:32.565523 22461 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:32.569063 22275 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.573256 22273 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:32.574988 22461 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:32.586619 22437 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.597997 22273 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.617048 21549 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.621461 21453 catalog_manager.cc:5526] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.649870 22406 raft_consensus.cc:1059] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:32.638888 21630 raft_consensus.cc:2949] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.653483 22406 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:32.666239 21698 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:32.674213 22274 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:32.675789 22461 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:32.676149 21627 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:32.685205 22473 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:32.711683 22387 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.719447 21624 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.722790 22274 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.724743 21698 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.734880 22459 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:32.743736 21440 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet cc5506492f514407a5097b5b2e8c7042 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250114 08:58:32.745627 22501 log.cc:926] Time spent T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250114 08:58:32.753286 21552 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:32.755120 21450 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.758132 22406 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:32.780216 22274 raft_consensus.cc:1212] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:32.784603 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5506492f514407a5097b5b2e8c7042 with delete_type TABLET_DATA_TOMBSTONED (TS 28a83b7a7d9d4aa899b425086aa45a8e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33868
I20250114 08:58:32.801671 22413 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:32.808616 21440 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 2e77833a006f478e86764142c505ab6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:32.850842 22504 tablet_replica.cc:331] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: stopping tablet replica
I20250114 08:58:32.859027 22272 raft_consensus.cc:2949] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.866446 21453 catalog_manager.cc:5526] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:32.880076 22413 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.884704 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77833a006f478e86764142c505ab6b with delete_type TABLET_DATA_TOMBSTONED (TS 28a83b7a7d9d4aa899b425086aa45a8e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33868
I20250114 08:58:32.899230 21698 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.910143 22504 raft_consensus.cc:2238] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:32.921080 22504 raft_consensus.cc:2267] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:32.922152 21451 catalog_manager.cc:5526] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.925664 21552 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.949601 22272 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:32.962430 22504 ts_tablet_manager.cc:1905] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:32.985343 21627 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:33.002002 22272 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:33.004890 22459 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:33.015480 21552 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:33.041220 22440 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.044852 22272 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.051913 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f21ba5689065467b967a5802582959dd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:33.052513 22459 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:33.071554 21451 catalog_manager.cc:5526] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.097988 21694 tablet_service.cc:1514] Processing DeleteTablet for tablet f21ba5689065467b967a5802582959dd with delete_type TABLET_DATA_TOMBSTONED (TS cec0ef3bbb654988bf2123d8eef7a0a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46386
I20250114 08:58:33.100661 21552 raft_consensus.cc:2949] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.114146 22506 tablet_replica.cc:331] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: stopping tablet replica
I20250114 08:58:33.219837 22504 ts_tablet_manager.cc:1918] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:33.220372 22504 log.cc:1198] T cc5506492f514407a5097b5b2e8c7042 P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/wals/cc5506492f514407a5097b5b2e8c7042
I20250114 08:58:33.222121 22504 tablet_replica.cc:331] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: stopping tablet replica
I20250114 08:58:33.225425 22504 raft_consensus.cc:2238] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:33.225883 21438 catalog_manager.cc:4872] TS 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): tablet cc5506492f514407a5097b5b2e8c7042 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:33.236549 22506 raft_consensus.cc:2238] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:33.236816 22504 raft_consensus.cc:2267] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:33.237324 22506 raft_consensus.cc:2267] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:33.242439 22504 ts_tablet_manager.cc:1905] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:33.242862 22506 ts_tablet_manager.cc:1905] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:33.272871 22504 ts_tablet_manager.cc:1918] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:33.273478 22504 log.cc:1198] T 2e77833a006f478e86764142c505ab6b P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/wals/2e77833a006f478e86764142c505ab6b
I20250114 08:58:33.291682 22458 raft_consensus.cc:1059] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:33.294227 22458 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:33.300585 21438 catalog_manager.cc:4872] TS 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): tablet 2e77833a006f478e86764142c505ab6b (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:33.311448 22272 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:33.313825 22458 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:33.316069 21698 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:33.317710 22466 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:33.323529 21627 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:33.331905 22333 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:33.367403 22506 ts_tablet_manager.cc:1918] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:33.368032 22506 log.cc:1198] T f21ba5689065467b967a5802582959dd P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root/wals/f21ba5689065467b967a5802582959dd
I20250114 08:58:33.366827 22508 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.370412 21440 catalog_manager.cc:4872] TS cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): tablet f21ba5689065467b967a5802582959dd (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:33.371518 22272 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.381623 21698 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.389039 21627 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.387341 21453 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.441886 21552 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:33.451231 22272 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:33.451434 21627 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:33.454502 22333 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:33.460856 22333 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:33.466392 22508 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.469858 22275 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.494606 21627 raft_consensus.cc:2949] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.500344 21438 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d19fa6dec61d44618fee387c9b1f2994 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:33.498620 21453 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:33.542755 21694 tablet_service.cc:1514] Processing DeleteTablet for tablet d19fa6dec61d44618fee387c9b1f2994 with delete_type TABLET_DATA_TOMBSTONED (TS cec0ef3bbb654988bf2123d8eef7a0a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46386
I20250114 08:58:33.543910 22506 tablet_replica.cc:331] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: stopping tablet replica
I20250114 08:58:33.546921 22506 raft_consensus.cc:2238] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:33.547832 22506 raft_consensus.cc:2267] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:33.559796 22506 ts_tablet_manager.cc:1905] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:33.578725 22506 ts_tablet_manager.cc:1918] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:33.579282 22506 log.cc:1198] T d19fa6dec61d44618fee387c9b1f2994 P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root/wals/d19fa6dec61d44618fee387c9b1f2994
I20250114 08:58:33.581043 21440 catalog_manager.cc:4872] TS cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): tablet d19fa6dec61d44618fee387c9b1f2994 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:34.673835 21698 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
W20250114 08:58:34.694751 21667 consensus_peers.cc:487] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89ff7b2c5f4e465fb847cd8b1c0b9e5a. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:34.698612 21548 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:34.700081 22512 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:34.701891 21627 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:34.703495 22512 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:34.736791 22387 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.742744 21624 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.746274 21552 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.760676 21451 catalog_manager.cc:5526] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.792513 21630 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
W20250114 08:58:34.820293 21593 consensus_peers.cc:487] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc9f4dcde58e4d508d5ffbbf301a7a6b. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:34.821774 21698 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:34.823458 22459 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:34.825973 21552 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:34.827720 22459 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 08:58:34.852612 22443 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.861068 21696 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.868559 21554 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:34.908816 21548 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:34.923288 21450 catalog_manager.cc:5526] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:34.945621 21696 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:34.947273 22356 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 08:58:34.977658 21518 consensus_peers.cc:487] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f466e3efd6cb41329852e9028490bdf7. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:35.027477 22528 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
W20250114 08:58:35.029170 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 50485.8us
I20250114 08:58:35.029655 21696 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.035152 21630 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:35.043498 22508 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 08:58:35.053762 21630 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.058338 21624 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.073930 21698 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250114 08:58:35.074335 21593 consensus_peers.cc:487] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abedb4afaad04af9bbbb7d4266fd2632. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:35.075848 22529 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:35.079902 21548 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:35.082372 22529 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:35.075788 21453 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.101611 22529 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.111076 21698 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.171514 21450 catalog_manager.cc:5526] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.183581 21554 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:35.248327 22534 ts_tablet_manager.cc:927] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:35.252444 22534 tablet_copy_client.cc:323] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.131:37485
I20250114 08:58:35.253994 21714 tablet_copy_service.cc:140] P cec0ef3bbb654988bf2123d8eef7a0a7: Received BeginTabletCopySession request for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:45060)
I20250114 08:58:35.254590 21714 tablet_copy_service.cc:161] P cec0ef3bbb654988bf2123d8eef7a0a7: Beginning new tablet copy session on tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:45060: session id = 57f9cc1dd3c64647848a33a65b08c893-89ff7b2c5f4e465fb847cd8b1c0b9e5a
I20250114 08:58:35.275650 21714 tablet_copy_source_session.cc:215] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:35.279094 22534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:35.329407 22534 tablet_copy_client.cc:806] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:35.330219 22534 tablet_copy_client.cc:670] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:35.335134 22534 tablet_copy_client.cc:538] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:35.354672 22534 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:35.406589 22534 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:35.409335 22534 tablet_bootstrap.cc:492] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:35.410311 22534 ts_tablet_manager.cc:1397] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.056s user 0.024s sys 0.013s
I20250114 08:58:35.413503 22534 raft_consensus.cc:357] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.414685 22534 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:35.415480 22534 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.425570 22537 ts_tablet_manager.cc:927] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:35.446523 22534 ts_tablet_manager.cc:1428] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.036s user 0.010s sys 0.001s
I20250114 08:58:35.451205 21714 tablet_copy_service.cc:342] P cec0ef3bbb654988bf2123d8eef7a0a7: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-89ff7b2c5f4e465fb847cd8b1c0b9e5a received from {username='slave'} at 127.0.0.1:45060
I20250114 08:58:35.451676 21714 tablet_copy_service.cc:434] P cec0ef3bbb654988bf2123d8eef7a0a7: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-89ff7b2c5f4e465fb847cd8b1c0b9e5a on tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:35.477732 22537 tablet_copy_client.cc:323] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:35.479530 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:35.480038 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet cc9f4dcde58e4d508d5ffbbf301a7a6b from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-cc9f4dcde58e4d508d5ffbbf301a7a6b
I20250114 08:58:35.488835 21640 tablet_copy_source_session.cc:215] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:35.492616 22537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:35.494179 22534 ts_tablet_manager.cc:927] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:35.513636 22534 tablet_copy_client.cc:323] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:35.521116 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet abedb4afaad04af9bbbb7d4266fd2632 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:35.521836 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet abedb4afaad04af9bbbb7d4266fd2632 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-abedb4afaad04af9bbbb7d4266fd2632
I20250114 08:58:35.561053 21640 tablet_copy_source_session.cc:215] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:35.564718 22534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abedb4afaad04af9bbbb7d4266fd2632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:35.577556 22541 ts_tablet_manager.cc:927] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:35.653985 22275 raft_consensus.cc:1212] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:35.666381 22534 tablet_copy_client.cc:806] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:35.666993 22534 tablet_copy_client.cc:670] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:35.667472 22537 tablet_copy_client.cc:806] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:35.668077 22537 tablet_copy_client.cc:670] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:35.678956 22534 tablet_copy_client.cc:538] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:35.680677 22537 tablet_copy_client.cc:538] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:35.696274 22541 tablet_copy_client.cc:323] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.129:45539
I20250114 08:58:35.700290 21565 tablet_copy_service.cc:140] P 28a83b7a7d9d4aa899b425086aa45a8e: Received BeginTabletCopySession request for tablet f466e3efd6cb41329852e9028490bdf7 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:50144)
I20250114 08:58:35.700881 21565 tablet_copy_service.cc:161] P 28a83b7a7d9d4aa899b425086aa45a8e: Beginning new tablet copy session on tablet f466e3efd6cb41329852e9028490bdf7 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:50144: session id = 57f9cc1dd3c64647848a33a65b08c893-f466e3efd6cb41329852e9028490bdf7
I20250114 08:58:35.696763 22534 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:35.709803 21565 tablet_copy_source_session.cc:215] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:35.713112 22537 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:35.715868 22541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f466e3efd6cb41329852e9028490bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:35.759459 22534 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:35.759954 22541 tablet_copy_client.cc:806] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:35.760596 22534 tablet_bootstrap.cc:492] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:35.760957 22541 tablet_copy_client.cc:670] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:35.761379 22534 ts_tablet_manager.cc:1397] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.065s user 0.018s sys 0.006s
I20250114 08:58:35.774243 22541 tablet_copy_client.cc:538] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:35.782116 22534 raft_consensus.cc:357] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.782857 22534 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:35.783445 22534 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.788641 22534 ts_tablet_manager.cc:1428] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250114 08:58:35.791244 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-abedb4afaad04af9bbbb7d4266fd2632 received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:35.791666 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-abedb4afaad04af9bbbb7d4266fd2632 on tablet abedb4afaad04af9bbbb7d4266fd2632 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:35.802668 22541 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
W20250114 08:58:35.865229 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 58537.8us
I20250114 08:58:35.866567 22537 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:35.867671 22537 tablet_bootstrap.cc:492] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:35.868757 22537 ts_tablet_manager.cc:1397] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.156s user 0.051s sys 0.016s
I20250114 08:58:35.872690 22537 raft_consensus.cc:357] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.876272 22537 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
W20250114 08:58:35.877522 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 72326us
I20250114 08:58:35.879726 22537 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.884174 22537 ts_tablet_manager.cc:1428] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20250114 08:58:35.887521 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-cc9f4dcde58e4d508d5ffbbf301a7a6b received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:35.887981 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-cc9f4dcde58e4d508d5ffbbf301a7a6b on tablet cc9f4dcde58e4d508d5ffbbf301a7a6b with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:35.907738 22275 raft_consensus.cc:1212] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:35.953570 22541 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:35.954723 22541 tablet_bootstrap.cc:492] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:35.955578 22541 ts_tablet_manager.cc:1397] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.153s user 0.037s sys 0.005s
I20250114 08:58:35.958738 22541 raft_consensus.cc:357] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.960222 22541 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:35.961189 22541 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:35.964160 22541 ts_tablet_manager.cc:1428] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20250114 08:58:35.966533 21565 tablet_copy_service.cc:342] P 28a83b7a7d9d4aa899b425086aa45a8e: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-f466e3efd6cb41329852e9028490bdf7 received from {username='slave'} at 127.0.0.1:50144
I20250114 08:58:35.966955 21565 tablet_copy_service.cc:434] P 28a83b7a7d9d4aa899b425086aa45a8e: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-f466e3efd6cb41329852e9028490bdf7 on tablet f466e3efd6cb41329852e9028490bdf7 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:36.023062 22275 raft_consensus.cc:1212] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:36.114660 22275 raft_consensus.cc:1212] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:36.198822 22387 raft_consensus.cc:1059] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:36.204000 22387 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:36.235435 21552 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.237346 22546 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:36.246927 22275 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.250236 22387 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250114 08:58:36.264520 21622 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.267297 22547 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:36.296432 22549 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.298198 21630 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.305121 21552 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.322647 22275 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.328167 21453 catalog_manager.cc:5526] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.391263 21696 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:36.397452 22275 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:36.399159 21552 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:36.403990 22547 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:36.413398 22546 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:36.415426 22459 raft_consensus.cc:1059] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:36.417743 22459 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:36.430415 22512 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.452972 22275 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.474915 22272 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.476589 22459 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:36.478770 21552 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.486439 22459 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:36.495872 21700 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.521499 22459 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:36.523609 22528 raft_consensus.cc:1059] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:36.526494 22528 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:36.540539 21440 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:36.550652 22274 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.552811 22508 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:36.559259 21696 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.556010 22516 raft_consensus.cc:1059] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:36.561347 21548 raft_consensus.cc:2949] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.562428 22516 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:36.569624 22528 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:36.556943 21624 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.588166 22443 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.594121 21452 catalog_manager.cc:5526] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New cstate: current_term: 1 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:36.601783 22272 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.623903 21618 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a with delete_type TABLET_DATA_TOMBSTONED (TS d7732324fb5f48d3b16a29d4bb2584ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57308
I20250114 08:58:36.634752 22528 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:36.669842 21452 catalog_manager.cc:5526] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.671980 22274 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.708912 22516 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:36.718529 22562 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.725853 21696 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.763792 21451 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:36.792971 22272 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.804558 21701 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.811538 21552 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:36.816974 21548 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.836659 22443 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:36.838800 22567 tablet_replica.cc:331] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: stopping tablet replica
I20250114 08:58:36.843961 22567 raft_consensus.cc:2238] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:36.844843 22567 raft_consensus.cc:2267] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:36.849284 22567 ts_tablet_manager.cc:1905] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:36.851990 21552 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:36.894379 22272 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:36.896248 22562 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:36.905967 21701 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.908145 22562 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.913640 22272 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:58:36.935389 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 73628.8us
I20250114 08:58:36.946651 22571 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:36.951215 21700 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:36.962527 21438 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f466e3efd6cb41329852e9028490bdf7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:36.964633 22508 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:36.965986 21452 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New cstate: current_term: 1 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.968449 22567 ts_tablet_manager.cc:1918] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20250114 08:58:36.973973 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 62721.3us
I20250114 08:58:36.974509 22567 log.cc:1198] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P d7732324fb5f48d3b16a29d4bb2584ed: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root/wals/89ff7b2c5f4e465fb847cd8b1c0b9e5a
I20250114 08:58:36.985221 21698 raft_consensus.cc:2949] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:36.998844 22571 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.006021 21618 tablet_service.cc:1514] Processing DeleteTablet for tablet f466e3efd6cb41329852e9028490bdf7 with delete_type TABLET_DATA_TOMBSTONED (TS d7732324fb5f48d3b16a29d4bb2584ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57308
I20250114 08:58:37.010160 22567 tablet_replica.cc:331] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: stopping tablet replica
I20250114 08:58:37.011416 22567 raft_consensus.cc:2238] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:37.012260 22567 pending_rounds.cc:62] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Trying to abort 1 pending ops.
I20250114 08:58:37.003571 21698 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.012630 22567 pending_rounds.cc:69] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7114117597285572608 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f466e3efd6cb41329852e9028490bdf7" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } } }
I20250114 08:58:37.025676 22567 raft_consensus.cc:2911] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20250114 08:58:37.026189 22567 raft_consensus.cc:2883] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250114 08:58:37.026598 21552 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.029963 21439 catalog_manager.cc:4872] TS d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): tablet 89ff7b2c5f4e465fb847cd8b1c0b9e5a (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:37.026664 22567 raft_consensus.cc:2267] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:37.042331 21624 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:37.041692 22272 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.053915 21451 catalog_manager.cc:5526] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:37.057665 22274 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:37.060664 22567 ts_tablet_manager.cc:1905] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:37.063531 22529 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:37.091028 22567 ts_tablet_manager.cc:1918] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:37.091599 22567 log.cc:1198] T f466e3efd6cb41329852e9028490bdf7 P d7732324fb5f48d3b16a29d4bb2584ed: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-1-root/wals/f466e3efd6cb41329852e9028490bdf7
I20250114 08:58:37.093772 21549 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:37.097466 22571 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:37.106519 22443 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.107594 21549 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.111721 22272 raft_consensus.cc:2949] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.160377 21439 catalog_manager.cc:4872] TS d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): tablet f466e3efd6cb41329852e9028490bdf7 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:37.185108 21621 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:37.193787 22272 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:37.195336 22570 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:58:37.206454 22571 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.210919 21451 catalog_manager.cc:5526] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.215044 21701 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250114 08:58:37.218727 22274 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.232959 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:37.245970 21694 tablet_service.cc:1514] Processing DeleteTablet for tablet abedb4afaad04af9bbbb7d4266fd2632 with delete_type TABLET_DATA_TOMBSTONED (TS cec0ef3bbb654988bf2123d8eef7a0a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46386
I20250114 08:58:37.265408 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet abedb4afaad04af9bbbb7d4266fd2632 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:58:37.300899 21451 catalog_manager.cc:5526] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.313030 22570 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:37.315444 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f4dcde58e4d508d5ffbbf301a7a6b with delete_type TABLET_DATA_TOMBSTONED (TS 28a83b7a7d9d4aa899b425086aa45a8e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33868
I20250114 08:58:37.338615 22580 tablet_replica.cc:331] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: stopping tablet replica
I20250114 08:58:37.342176 22580 raft_consensus.cc:2238] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:37.352262 21696 raft_consensus.cc:2949] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:37.399961 22580 raft_consensus.cc:2267] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:37.418761 22580 ts_tablet_manager.cc:1905] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:37.434386 22581 tablet_replica.cc:331] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: stopping tablet replica
I20250114 08:58:37.441881 22581 raft_consensus.cc:2238] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 08:58:37.447942 22581 raft_consensus.cc:2267] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 08:58:37.445391 22580 ts_tablet_manager.cc:1918] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:37.448807 22580 log.cc:1198] T abedb4afaad04af9bbbb7d4266fd2632 P cec0ef3bbb654988bf2123d8eef7a0a7: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-2-root/wals/abedb4afaad04af9bbbb7d4266fd2632
I20250114 08:58:37.450609 21440 catalog_manager.cc:4872] TS cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): tablet abedb4afaad04af9bbbb7d4266fd2632 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:58:37.455230 22581 ts_tablet_manager.cc:1905] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 08:58:37.472896 22581 ts_tablet_manager.cc:1918] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250114 08:58:37.473387 22581 log.cc:1198] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/wals/cc9f4dcde58e4d508d5ffbbf301a7a6b
I20250114 08:58:37.475078 21438 catalog_manager.cc:4872] TS 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): tablet cc9f4dcde58e4d508d5ffbbf301a7a6b (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
W20250114 08:58:38.321894 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 53401.1us
I20250114 08:58:38.445544 21622 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:38.454725 21698 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250114 08:58:38.455121 21593 consensus_peers.cc:487] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b369b35736c413ca069db061ed36bc5. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:38.457249 21552 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:38.455996 22570 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:38.459165 22591 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:38.468189 22584 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.470292 21549 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.484582 21698 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.496642 21622 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:38.502055 21451 catalog_manager.cc:5526] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
W20250114 08:58:38.511871 21593 consensus_peers.cc:487] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ada199404e61479593fa1b64c574bcb8. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:38.520507 21549 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:38.525276 21698 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:38.524070 22570 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250114 08:58:38.529239 22583 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:38.551651 22570 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.556531 21549 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.558769 21696 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.576041 21552 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:38.577229 21452 catalog_manager.cc:5526] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.589903 21696 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250114 08:58:38.590497 21518 consensus_peers.cc:487] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82452b55145444529bf0609cd7767adf. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:38.591431 22590 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:38.592628 21622 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:38.598656 22590 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:38.605708 21696 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.606164 22582 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.621505 21453 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.629360 21621 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.630527 21630 consensus_queue.cc:237] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
W20250114 08:58:38.646730 21593 consensus_peers.cc:487] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed -> Peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045): Couldn't send request to peer 57f9cc1dd3c64647848a33a65b08c893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3650a91ff4a4213b9091f197bf75fb9. This is attempt 1: this message will repeat every 5th retry.
I20250114 08:58:38.646898 21696 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:38.648802 22583 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:38.652990 21552 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 08:58:38.654606 22591 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:38.666119 22583 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.669607 21698 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:38.691680 21452 catalog_manager.cc:5526] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:38.692298 21549 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } } }
I20250114 08:58:39.040198 22611 ts_tablet_manager.cc:927] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:39.057981 22610 ts_tablet_manager.cc:927] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:39.063686 22610 tablet_copy_client.cc:323] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:39.066781 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet b3650a91ff4a4213b9091f197bf75fb9 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:39.068215 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet b3650a91ff4a4213b9091f197bf75fb9 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-b3650a91ff4a4213b9091f197bf75fb9
I20250114 08:58:39.079033 21640 tablet_copy_source_session.cc:215] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:39.101147 22610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3650a91ff4a4213b9091f197bf75fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:39.109056 22611 tablet_copy_client.cc:323] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:39.113627 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet 3b369b35736c413ca069db061ed36bc5 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:39.114231 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet 3b369b35736c413ca069db061ed36bc5 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-3b369b35736c413ca069db061ed36bc5
I20250114 08:58:39.128592 22614 ts_tablet_manager.cc:927] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:39.132220 21640 tablet_copy_source_session.cc:215] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:39.132930 22610 tablet_copy_client.cc:806] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:39.133634 22610 tablet_copy_client.cc:670] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:39.136358 22611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b369b35736c413ca069db061ed36bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:39.138062 22615 ts_tablet_manager.cc:927] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Initiating tablet copy from peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:39.143353 22610 tablet_copy_client.cc:538] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:39.153316 22610 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:39.156869 22614 tablet_copy_client.cc:323] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.129:45539
I20250114 08:58:39.158483 21565 tablet_copy_service.cc:140] P 28a83b7a7d9d4aa899b425086aa45a8e: Received BeginTabletCopySession request for tablet 82452b55145444529bf0609cd7767adf from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:50144)
I20250114 08:58:39.159034 21565 tablet_copy_service.cc:161] P 28a83b7a7d9d4aa899b425086aa45a8e: Beginning new tablet copy session on tablet 82452b55145444529bf0609cd7767adf from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:50144: session id = 57f9cc1dd3c64647848a33a65b08c893-82452b55145444529bf0609cd7767adf
I20250114 08:58:39.160861 22615 tablet_copy_client.cc:323] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.130:36891
I20250114 08:58:39.162590 21640 tablet_copy_service.cc:140] P d7732324fb5f48d3b16a29d4bb2584ed: Received BeginTabletCopySession request for tablet ada199404e61479593fa1b64c574bcb8 from peer 57f9cc1dd3c64647848a33a65b08c893 ({username='slave'} at 127.0.0.1:43436)
I20250114 08:58:39.163117 21640 tablet_copy_service.cc:161] P d7732324fb5f48d3b16a29d4bb2584ed: Beginning new tablet copy session on tablet ada199404e61479593fa1b64c574bcb8 from peer 57f9cc1dd3c64647848a33a65b08c893 at {username='slave'} at 127.0.0.1:43436: session id = 57f9cc1dd3c64647848a33a65b08c893-ada199404e61479593fa1b64c574bcb8
I20250114 08:58:39.164861 22611 tablet_copy_client.cc:806] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:39.165436 22611 tablet_copy_client.cc:670] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:39.167832 21565 tablet_copy_source_session.cc:215] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:39.170836 21640 tablet_copy_source_session.cc:215] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Tablet Copy: opened 0 blocks and 1 log segments
I20250114 08:58:39.172115 22614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82452b55145444529bf0609cd7767adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:39.171864 22611 tablet_copy_client.cc:538] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:39.174316 22615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada199404e61479593fa1b64c574bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 08:58:39.193604 22611 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:39.195984 22614 tablet_copy_client.cc:806] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:39.197849 22614 tablet_copy_client.cc:670] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
W20250114 08:58:39.206111 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002cc760 after lost signal to thread 20385
I20250114 08:58:39.205981 22614 tablet_copy_client.cc:538] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250114 08:58:39.207110 22296 debug-util.cc:398] Leaking SignalData structure 0x7b0800315e60 after lost signal to thread 21487
W20250114 08:58:39.208274 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1ae0 after lost signal to thread 21579
W20250114 08:58:39.209275 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1280 after lost signal to thread 21654
W20250114 08:58:39.210283 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe0c0 after lost signal to thread 21728
W20250114 08:58:39.211395 22296 debug-util.cc:398] Leaking SignalData structure 0x7b08002c7520 after lost signal to thread 22299
I20250114 08:58:39.227401 22615 tablet_copy_client.cc:806] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 0 data blocks...
I20250114 08:58:39.228281 22615 tablet_copy_client.cc:670] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Starting download of 1 WAL segments...
I20250114 08:58:39.232828 22615 tablet_copy_client.cc:538] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 08:58:39.250046 22614 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:39.252823 22615 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap starting.
I20250114 08:58:39.256453 22610 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:39.257652 22610 tablet_bootstrap.cc:492] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:39.259816 22610 ts_tablet_manager.cc:1397] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.107s user 0.015s sys 0.020s
I20250114 08:58:39.299882 22610 raft_consensus.cc:357] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.300972 22610 raft_consensus.cc:738] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:39.306639 22611 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:39.309233 22611 tablet_bootstrap.cc:492] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:39.310050 22611 ts_tablet_manager.cc:1397] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.034s
I20250114 08:58:39.313505 22611 raft_consensus.cc:357] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.306430 22610 consensus_queue.cc:260] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.321971 22611 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:39.323835 22611 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.345166 22611 ts_tablet_manager.cc:1428] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.034s user 0.009s sys 0.001s
I20250114 08:58:39.350855 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-3b369b35736c413ca069db061ed36bc5 received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:39.351446 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-3b369b35736c413ca069db061ed36bc5 on tablet 3b369b35736c413ca069db061ed36bc5 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:39.387408 22614 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:39.388584 22614 tablet_bootstrap.cc:492] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:39.389405 22614 ts_tablet_manager.cc:1397] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.140s user 0.023s sys 0.010s
I20250114 08:58:39.400110 22614 raft_consensus.cc:357] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.401067 22614 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:39.401966 22614 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.404924 22614 ts_tablet_manager.cc:1428] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250114 08:58:39.415941 21565 tablet_copy_service.cc:342] P 28a83b7a7d9d4aa899b425086aa45a8e: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-82452b55145444529bf0609cd7767adf received from {username='slave'} at 127.0.0.1:50144
I20250114 08:58:39.416431 21565 tablet_copy_service.cc:434] P 28a83b7a7d9d4aa899b425086aa45a8e: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-82452b55145444529bf0609cd7767adf on tablet 82452b55145444529bf0609cd7767adf with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:39.494218 22610 ts_tablet_manager.cc:1428] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.230s user 0.027s sys 0.050s
I20250114 08:58:39.497160 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-b3650a91ff4a4213b9091f197bf75fb9 received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:39.497694 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-b3650a91ff4a4213b9091f197bf75fb9 on tablet b3650a91ff4a4213b9091f197bf75fb9 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:39.522848 22615 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 08:58:39.526047 22615 tablet_bootstrap.cc:492] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Bootstrap complete.
I20250114 08:58:39.526993 22615 ts_tablet_manager.cc:1397] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Time spent bootstrapping tablet: real 0.274s user 0.047s sys 0.040s
I20250114 08:58:39.531173 22615 raft_consensus.cc:357] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.532527 22615 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Initialized, Role: LEARNER
I20250114 08:58:39.533313 22615 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:39.539578 22615 ts_tablet_manager.cc:1428] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250114 08:58:39.541620 21640 tablet_copy_service.cc:342] P d7732324fb5f48d3b16a29d4bb2584ed: Request end of tablet copy session 57f9cc1dd3c64647848a33a65b08c893-ada199404e61479593fa1b64c574bcb8 received from {username='slave'} at 127.0.0.1:43436
I20250114 08:58:39.542129 21640 tablet_copy_service.cc:434] P d7732324fb5f48d3b16a29d4bb2584ed: ending tablet copy session 57f9cc1dd3c64647848a33a65b08c893-ada199404e61479593fa1b64c574bcb8 on tablet ada199404e61479593fa1b64c574bcb8 with peer 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:39.557119 22272 raft_consensus.cc:1212] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:39.571627 22272 raft_consensus.cc:1212] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:39.621469 22272 raft_consensus.cc:1212] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250114 08:58:39.638407 22272 raft_consensus.cc:1212] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250114 08:58:40.091158 22626 raft_consensus.cc:1059] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:40.094169 22626 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:40.108379 22274 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.110652 22624 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:40.125453 21698 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.127139 22584 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:40.130275 22625 raft_consensus.cc:1059] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:40.131709 21550 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.133901 22626 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:40.151504 22583 raft_consensus.cc:1059] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:40.152745 22591 raft_consensus.cc:1059] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: attempting to promote NON_VOTER 57f9cc1dd3c64647848a33a65b08c893 to VOTER
I20250114 08:58:40.154201 22625 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:40.155300 22591 consensus_queue.cc:237] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:40.159092 22583 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:40.179407 22274 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.182825 21552 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.189942 22626 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:40.191107 22624 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:40.232679 22626 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:40.238520 21554 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:40.267879 22272 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:40.284480 22275 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEARNER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:40.296519 21549 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.298305 21452 catalog_manager.cc:5526] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:40.306542 22275 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 LEARNER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.306879 22625 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:40.421597 21698 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20250114 08:58:40.422071 22628 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.077s user 0.009s sys 0.006s
I20250114 08:58:40.430274 21696 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.433565 21697 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 08:58:40.354030 22624 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
W20250114 08:58:40.448601 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 119778us
I20250114 08:58:40.443120 22582 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250114 08:58:40.465353 22583 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
W20250114 08:58:40.503011 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 216222us
I20250114 08:58:41.074604 22648 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:41.429805 22648 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:41.549281 22648 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.601662 22659 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.602557 22659 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.620059 22655 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:41.620255 22653 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:41.620839 22655 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:41.621039 22653 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:41.623482 22655 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.624631 22653 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.802248 22661 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:41.802979 22661 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.805963 22661 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.807114 22653 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.810287 22659 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.903739 22661 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.903733 22653 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.904848 22661 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.907277 22653 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:41.908071 22659 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.908869 22659 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.624133 22655 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.910899 22655 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:41.913690 22655 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.915654 22659 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.916374 22659 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:41.916976 22659 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.919557 22659 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:41.842573 22660 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:41.920943 22660 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.874784 22663 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:41.924706 22663 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.927974 22663 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.888357 22656 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.889685 22658 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.930440 22656 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.931469 22658 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.934171 22661 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.934904 22658 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.941581 22660 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.942807 22656 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.951418 22666 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:41.901911 22664 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:41.954205 22664 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.952281 22666 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:41.957492 22664 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:41.957768 22666 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.018975 22655 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.023615 22666 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.022374 22655 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.024883 22666 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.028740 22655 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.025436 22664 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.029346 22666 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.030005 22664 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.033135 22664 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.159303 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.162825 22591 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.164094 22591 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.166390 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.167644 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.168169 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.169173 21552 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:42.169559 22591 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.170243 22591 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.170949 22591 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.173125 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.174134 21550 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:42.181600 22591 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.183787 22583 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.184721 22583 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.186132 22591 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.186990 22591 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.187518 22583 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.188149 22583 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.188827 22583 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.189482 22591 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.190112 22591 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.190819 22591 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.191324 22583 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.193118 22669 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.193696 22583 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.193908 22669 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.194545 22583 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.195911 22646 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.196394 22669 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.196771 22583 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.196628 22646 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.196988 22669 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.197793 22669 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.199980 22669 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.200311 22591 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.203903 22653 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.204834 22653 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.206701 22646 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.217121 22653 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:42.217509 21669 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.775s user 0.003s sys 0.073s
W20250114 08:58:42.218045 21669 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.775s user 0.003s sys 0.073s
I20250114 08:58:42.221242 22653 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.222383 22653 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.225633 22653 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.233736 22670 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.237190 22670 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.242931 22655 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.243834 22655 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.246654 22655 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:42.247404 22673 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.244439 22670 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.249151 22670 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.250012 22670 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.251407 22673 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:42.252645 22655 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.253351 22655 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.254245 22673 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.255527 22655 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.259500 22670 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:42.267176 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.267395 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.268904 22671 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.273272 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.274411 21622 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
W20250114 08:58:42.275337 21519 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.769s user 0.010s sys 0.015s
W20250114 08:58:42.275820 21519 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.770s user 0.010s sys 0.015s
I20250114 08:58:42.275868 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.268993 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.270007 22671 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.287689 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.288936 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.291211 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.292387 22624 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.293172 22624 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.294287 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.295259 21629 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.295466 22624 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.305600 22671 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:42.348099 21593 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.853s user 0.022s sys 0.054s
W20250114 08:58:42.350528 21593 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.856s user 0.024s sys 0.054s
I20250114 08:58:42.350838 22672 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.349318 21622 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:42.354584 22670 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.355403 22670 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:42.360318 22664 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.361115 22664 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.364799 21668 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:42.365870 22655 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:42.366258 22655 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.366629 22655 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:42.370172 21668 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:42.373423 22664 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.351716 22672 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.392585 22669 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.393657 22669 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.384625 22583 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.404734 22583 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:42.407666 22583 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:42.393682 21452 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 195) took 2120 ms. Trace:
I20250114 08:58:42.413161 21452 rpcz_store.cc:276] 0114 08:58:40.272884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:40.273089 (+ 205us) service_pool.cc:225] Handling call
0114 08:58:42.393377 (+2120288us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:40.422652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:40.423188 (+ 536us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:40.423218 (+ 30us) write_op.cc:435] Acquired schema lock
0114 08:58:40.423236 (+ 18us) tablet.cc:659] Decoding operations
0114 08:58:40.423559 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:40.423636 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:40.423663 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0114 08:58:40.423866 (+ 203us) tablet.cc:698] Row locks acquired
0114 08:58:40.423880 (+ 14us) write_op.cc:260] PREPARE: finished
0114 08:58:40.424037 (+ 157us) write_op.cc:270] Start()
0114 08:58:40.424135 (+ 98us) write_op.cc:276] Timestamp: P: 1736845120423987 usec, L: 0
0114 08:58:40.424158 (+ 23us) op_driver.cc:348] REPLICATION: starting
0114 08:58:40.433114 (+ 8956us) log.cc:844] Serialized 602 byte log entry
0114 08:58:41.485527 (+1052413us) op_driver.cc:464] REPLICATION: finished
0114 08:58:41.627218 (+141691us) write_op.cc:301] APPLY: starting
0114 08:58:41.627320 (+ 102us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:41.627441 (+ 121us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:41.627588 (+ 147us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:41.627849 (+ 261us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:41.628044 (+ 195us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 08:58:41.628120 (+ 76us) write_op.cc:312] APPLY: finished
0114 08:58:41.628351 (+ 231us) log.cc:844] Serialized 22 byte log entry
0114 08:58:41.629299 (+ 948us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:41.629404 (+ 105us) write_op.cc:454] Released schema lock
0114 08:58:41.629453 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:41.632411 (+ 2958us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:41.632456 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":141383,"apply.run_cpu_time_us":1889,"apply.run_wall_time_us":2616,"num_ops":1,"prepare.queue_time_us":119878,"prepare.run_cpu_time_us":14874,"prepare.run_wall_time_us":45312,"raft.queue_time_us":938232,"raft.run_cpu_time_us":11948,"raft.run_wall_time_us":212503,"replication_time_us":1061143,"thread_start_us":636582,"threads_started":4,"wal-append.queue_time_us":43877}]]}
I20250114 08:58:42.411792 22240 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:42.378372 22671 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.430385 22673 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.435873 22673 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.437585 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:42.438377 22272 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
W20250114 08:58:42.441149 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 72896.4us
I20250114 08:58:42.442281 22672 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.426585 22670 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.459308 22680 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.461494 22582 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.462400 22582 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.462531 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:42.464670 22664 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:42.465255 22664 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.431788 22669 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.465765 22664 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:42.468048 22678 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.468791 22678 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.472254 22666 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:42.472887 22666 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.477491 22666 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:42.477560 22677 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.431857 22583 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.482821 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:42.476204 22646 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.489537 22681 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.490213 22681 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.492141 22681 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.478395 22582 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.487530 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:42.499709 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.484690 22675 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:42.503105 22275 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:42.497819 22673 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.479882 22671 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.508450 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.460438 22680 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.478391 22677 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.501439 22273 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:42.502508 22582 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.516213 21629 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:42.506222 22584 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.517326 22667 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.507711 22675 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.511680 22669 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.501711 22646 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.521963 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.526649 21629 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:42.527694 22584 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.525622 22675 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:42.530663 22584 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.530912 22677 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.531595 22677 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.532300 22677 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.532977 22582 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.534868 22677 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.535395 22582 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:42.532999 22667 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.536688 22669 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.519819 22681 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.540563 21667 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:42.507179 22583 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.558486 21453 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 68) took 2111 ms. Trace:
I20250114 08:58:42.558621 22582 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.559078 21453 rpcz_store.cc:276] 0114 08:58:40.441718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:40.442434 (+ 716us) service_pool.cc:225] Handling call
0114 08:58:42.147641 (+1705207us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54001be754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f552d50 00007fa63f5d5f2a 00007fa6452f1dec 00007fa6453e9dee 00007fa64393cfc4 00007fa64393cf52 00007fa64393ce81 00007fa64393cdfc 00007fa64393c0b2 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:42.552792 (+405151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":252413,"reported_tablets":1,"spinlock_wait_cycles":3603896192}
I20250114 08:58:42.563817 22676 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:42.564348 22676 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.564769 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55864 (request call id 190) took 2227 ms. Trace:
W20250114 08:58:42.565030 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 66377.6us
I20250114 08:58:42.565189 21451 rpcz_store.cc:276] 0114 08:58:40.435678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:40.435818 (+ 140us) service_pool.cc:225] Handling call
0114 08:58:42.243876 (+1808058us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54001be754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f552d50 00007fa63f5d5f2a 00007fa6452f1dec 00007fa6453e9dee 00007fa64393cfc4 00007fa64393cf52 00007fa64393ce81 00007fa64393cdfc 00007fa64393c0b2 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:42.556182 (+312306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":160105,"reported_tablets":1,"spinlock_wait_cycles":3922255488}
I20250114 08:58:42.564819 22676 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:42.570449 22678 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.571501 21667 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:42.572703 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.572990 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.517814 22679 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.576733 22670 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:42.577303 22670 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.577768 22670 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:42.576396 22679 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.592020 21667 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:42.573514 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.606634 22681 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.614244 22591 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:42.617199 22681 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.619450 22679 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.620630 22669 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.621141 22686 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.621919 22583 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.621887 22686 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.623620 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.559430 22582 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.626793 22673 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:42.620700 22685 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.644801 22667 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.647439 22674 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.627344 22672 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.663275 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:42.663219 22674 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.676566 22672 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.656229 22691 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.641959 22673 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:58:42.683475 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 91588.9us
I20250114 08:58:42.661772 22583 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.679311 22653 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.681263 22667 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.651782 22669 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.698257 22590 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.698423 22688 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.688742 22673 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:42.684492 22671 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.702308 22645 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.706931 22653 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.701251 22646 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:42.697366 22680 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.699628 22681 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.715448 22684 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.726451 22625 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.727370 22653 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.710072 22645 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.723884 22678 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.733543 22694 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.713510 22667 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.740569 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.722532 22583 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.750501 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.733853 22272 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:58:42.752040 22689 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20250114 08:58:42.732295 22674 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.727154 22625 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.737174 22686 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.713109 22669 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.762782 22697 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:42.701153 22685 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.769565 21555 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.734658 22681 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.695313 22691 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.773268 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.738464 22653 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.766360 22582 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.785233 22683 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.770210 22682 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.771807 22674 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.726930 22684 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:42.730463 22590 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.794631 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.795507 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.797478 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.773048 22686 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.769068 22678 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.770246 22673 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.783128 22672 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.781054 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.773357 22646 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250114 08:58:42.802517 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.757812 22671 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250114 08:58:42.729521 22688 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.788492 22582 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.809331 22686 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.811734 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.810613 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.815938 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.816339 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.802287 22679 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.788754 22653 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.785866 22683 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.780812 22694 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.800952 22674 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.813093 21550 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:42.835947 22688 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.810231 22682 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.823836 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.806349 22696 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.851349 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.847534 22655 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.852361 21700 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.853825 22655 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.854151 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.854427 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.822495 22590 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.856094 21555 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.864912 21551 raft_consensus.cc:2405] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20250114 08:58:42.795599 22697 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.872326 22687 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.874740 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.857297 22669 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.828121 22679 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.873037 22687 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.880055 22668 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.886343 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.855037 22582 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.870018 22583 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.869459 22590 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.828080 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:42.887751 21552 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.892242 21700 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.869817 22697 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.902204 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.902330 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.903157 21552 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:42.851441 22696 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.908385 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.887895 22645 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.895211 22653 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.908385 22684 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.912403 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.913333 22691 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:42.913507 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.914302 21698 raft_consensus.cc:2388] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:42.916121 22684 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:42.902838 21553 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:42.898226 22625 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.897807 22677 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:42.916761 22684 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.920051 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:42.901110 22673 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:42.921222 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:42.941504 22694 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.908373 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:42.909544 22678 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:42.942173 22666 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.951349 22273 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:42.968916 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 84231.5us
I20250114 08:58:42.900560 22590 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:42.970629 22685 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:42.931504 22668 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.921267 22664 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:42.968919 22683 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.008553 22690 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:43.013461 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.010838 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.017442 22696 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.017469 22667 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.018359 22670 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.018858 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.019630 21696 raft_consensus.cc:2388] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:43.019603 21700 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:43.015758 21548 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:42.969290 22677 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.021996 22677 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.023304 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.032661 22275 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.057024 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.014973 22590 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.030491 22270 raft_consensus.cc:1235] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.072813 22697 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.056232 22679 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.055379 22684 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.015424 22682 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.030388 22685 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.055900 22582 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.030651 22673 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.076020 22690 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.078744 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.082041 22666 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.083187 22705 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.084322 22694 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.092415 22703 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.092865 22694 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.084885 22688 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.086424 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.074523 22670 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.054911 21550 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:43.057555 22674 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.087844 22653 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.091548 21552 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:43.030959 22687 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.029439 22676 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.092993 22703 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.101279 22668 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.097394 22688 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.102396 22273 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:43.103353 21697 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:43.100497 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.105379 21548 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:43.090718 22696 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.106792 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.107443 22664 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:43.107944 22675 consensus_meta.cc:287] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.579s user 0.018s sys 0.001s
I20250114 08:58:43.110316 22688 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.111609 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.103343 22694 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.114302 21551 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:43.089728 22697 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.108973 22696 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.091919 22685 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.105135 22653 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.120991 22696 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.093384 22705 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.123181 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.110872 22675 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.099536 22673 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:43.129722 21629 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.124892 22697 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.124900 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.129024 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.132324 22703 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.135525 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.137286 21701 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.125913 22686 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.143343 22705 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.143142 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.130961 22676 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.146706 22681 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.149987 22690 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.148725 22664 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.150012 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.150743 22664 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.151188 22664 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.151518 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.152350 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.153146 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.153853 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.155751 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.156389 21552 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:43.157218 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:43.157743 21552 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.161073 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.159368 22664 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.161688 21699 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.161748 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.162338 21703 raft_consensus.cc:1235] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.163012 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.163312 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.163026 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.165225 21695 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.165284 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:43.165877 22664 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.166184 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.166494 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.167060 21629 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:43.167383 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.148378 22653 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.171043 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.171044 22240 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:43.173650 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.174281 22675 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.174484 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.174957 22664 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:43.174824 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:43.175684 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 84389.9us
I20250114 08:58:43.176100 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.177242 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.177361 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.177862 21553 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:43.178151 22239 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.178669 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.179386 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.178611 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.181519 22685 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:43.181470 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:43.187884 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.187841 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.182338 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.188881 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.190902 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.193490 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.193761 21696 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:43.194084 21554 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:43.219317 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.219568 21629 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:43.220008 21725 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0340 after lost signal to thread 20385
I20250114 08:58:43.220258 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.220626 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:43.221135 21725 debug-util.cc:398] Leaking SignalData structure 0x7b0800200aa0 after lost signal to thread 21487
W20250114 08:58:43.222440 21725 debug-util.cc:398] Leaking SignalData structure 0x7b08002b9fc0 after lost signal to thread 21579
I20250114 08:58:43.222919 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:43.175711 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:43.223528 21725 debug-util.cc:398] Leaking SignalData structure 0x7b08002eaea0 after lost signal to thread 21654
W20250114 08:58:43.223440 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 52738.7us
I20250114 08:58:43.227149 22673 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:43.223485 21554 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.229526 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:43.230595 22685 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:43.231050 21669 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:43.231073 22685 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.231920 22685 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:43.232374 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.232689 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.233045 21704 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.221102 21701 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.234362 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.219193 21555 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:43.235574 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.242851 21703 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.243922 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:43.238322 21725 debug-util.cc:398] Leaking SignalData structure 0x7b08002e14c0 after lost signal to thread 21728
I20250114 08:58:43.245183 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:43.245453 21725 debug-util.cc:398] Leaking SignalData structure 0x7b08002890e0 after lost signal to thread 22299
I20250114 08:58:43.246297 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.247422 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.233345 21697 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.249164 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.250406 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.235677 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.235914 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.252223 21552 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:43.189195 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.254967 21554 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:43.257828 21551 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.191599 22664 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:43.260345 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.260694 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.261837 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.262976 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.263960 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.264807 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.264887 21521 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.265489 21697 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.266304 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.267195 22686 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.267403 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.267629 22686 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.268332 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.268601 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.267632 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.269292 21698 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.269778 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.271040 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.271298 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.261617 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.272521 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:43.272636 21704 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.266528 21521 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.274255 22682 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.274628 22682 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.275038 22682 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.275126 22582 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:43.276261 22582 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:43.276688 22582 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:43.276762 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.277599 22582 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.278802 22582 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.274639 22681 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.274791 22690 consensus_queue.cc:1046] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:43.280961 21521 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.268534 22686 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.281719 22687 raft_consensus.cc:3049] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:43.281816 22690 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.282152 22687 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:43.282303 22690 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.282720 22690 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.283202 22687 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.261855 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.280982 22681 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:43.285315 22681 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.285351 21621 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.284809 22687 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.286859 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.290287 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.290755 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.233063 22673 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.291461 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.294819 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.291733 21552 raft_consensus.cc:1235] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:43.292002 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 59677.2us
I20250114 08:58:43.296079 21669 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.296893 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.288151 21695 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.312386 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.312769 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.310647 21668 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.312928 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.313581 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.260401 22712 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:43.315512 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.296963 22673 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.320031 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.321645 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.324362 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.324859 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.325114 21621 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.325501 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.327667 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.306788 22653 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:43.329171 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.325481 21622 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.316008 22666 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:43.320326 22673 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.317912 22703 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:43.332418 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.330123 21521 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.333436 22699 consensus_queue.cc:1046] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 08:58:43.333473 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:43.322252 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 142934us
I20250114 08:58:43.308055 22684 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250114 08:58:43.347534 22684 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:43.347903 22684 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:43.348766 22684 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.340982 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.322045 22712 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.339192 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.320442 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:43.346472 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.345968 22708 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.353401 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.352619 22712 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:43.353271 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.357595 22241 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:43.353029 22670 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.358565 22708 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.353474 22239 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.353600 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.359005 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.307781 22664 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.360214 22670 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.367133 22713 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:43.367024 22693 consensus_queue.cc:1046] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250114 08:58:43.350438 22682 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.372207 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.360105 22699 raft_consensus.cc:3049] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
W20250114 08:58:43.388854 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 47941.6us
I20250114 08:58:43.408433 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.388427 22708 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.428215 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:43.418841 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 66396.3us
I20250114 08:58:43.424777 21521 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.425865 22713 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.426236 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.427095 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.430622 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.424563 21621 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:43.430626 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.388692 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.416505 22674 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.435045 21668 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.435936 21668 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.437435 21668 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.418315 22670 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.439419 21549 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.439657 22675 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.439949 21668 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.443338 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.444281 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:43.417691 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 92511.2us
I20250114 08:58:43.447377 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.424115 22699 raft_consensus.cc:738] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:43.448133 22693 raft_consensus.cc:3049] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:43.431034 22713 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.454180 21696 raft_consensus.cc:1235] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.454619 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.438730 22674 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.443032 22694 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.445444 22703 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.449205 22697 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.469950 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.448998 22706 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.522998 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.469977 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.469059 22679 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:43.472183 21521 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.471200 22674 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.442781 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:43.529633 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57508.6us
I20250114 08:58:43.445753 22675 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.472744 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.474265 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.471381 22694 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.456630 22713 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.479233 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.451900 22693 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:43.532040 22694 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.532763 22693 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.533995 22693 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.471540 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:43.525095 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 55318.6us
I20250114 08:58:43.481880 22697 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.445695 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.470546 22682 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.473318 22703 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.530831 21697 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:43.536489 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.537165 21521 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.526563 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.525648 22673 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:43.547096 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 78539.8us
I20250114 08:58:43.547096 22582 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:43.569082 21668 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:43.569217 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 99389.7us
I20250114 08:58:43.547488 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:43.570986 21521 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:43.548394 22715 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.548676 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.573566 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.574764 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.575915 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.548887 22702 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.576648 22702 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.577078 22702 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.529623 21695 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:43.550985 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.551522 21703 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:43.553215 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.554349 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.540122 22717 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.556561 22703 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.558283 22670 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:43.558432 22699 consensus_queue.cc:260] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.560014 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.561585 22690 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.562757 21547 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:58:43.565498 22711 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.335s user 0.005s sys 0.000s
I20250114 08:58:43.523540 22706 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.566650 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.579303 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.528067 22590 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:43.591804 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.535094 22675 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.545514 22697 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.582664 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.586225 22582 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:43.586462 21521 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:43.588835 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.589000 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.600639 22583 consensus_queue.cc:1046] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 08:58:43.605187 22710 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.412s user 0.000s sys 0.010s
I20250114 08:58:43.590729 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:43.583551 22717 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.594067 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.596802 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.597136 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.607612 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.581897 22668 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.579697 21451 catalog_manager.cc:5526] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:43.606398 22681 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.578373 22715 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:58:43.626103 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 56022.2us
I20250114 08:58:43.596837 22706 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.632802 22704 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.581319 22719 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.641170 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:43.625615 22699 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.616694 22683 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:43.626098 22717 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.629671 22668 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:43.600179 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 246780us
I20250114 08:58:43.681679 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.645486 22694 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.650878 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.643323 22719 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.650003 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.707937 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.650059 22704 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:43.628907 22715 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.599095 22241 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:43.708436 21625 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.712360 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.712896 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.615341 22713 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.701748 22668 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.718415 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.711494 22704 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.720074 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:43.722441 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 137207us
I20250114 08:58:43.710481 22719 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.734447 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.734652 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.724339 22666 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:43.710757 22712 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.736327 22271 raft_consensus.cc:1235] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.749763 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:43.753121 21555 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.751281 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:43.750049 22685 consensus_meta.cc:287] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.518s user 0.010s sys 0.000s
I20250114 08:58:43.758285 22272 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:43.727186 22239 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:43.746238 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.763988 21697 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.769048 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.773784 22270 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:43.762454 22681 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.764796 22685 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:43.764097 22690 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.746536 22241 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:43.769083 22673 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.805718 22670 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:43.819051 22724 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20250114 08:58:43.757763 22694 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.827137 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:43.719938 22722 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:43.791543 22664 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.832826 21699 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20250114 08:58:43.831651 21623 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:43.828204 22723 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.842335 22591 raft_consensus.cc:3049] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:43.843194 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.837038 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.849896 22591 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:43.832989 22681 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.856439 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.832262 22712 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:43.856315 22591 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.859700 22583 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:43.834704 22664 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:43.742360 22679 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:43.914407 22591 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.865235 22667 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:58:43.858394 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 139725us
I20250114 08:58:43.927359 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.928884 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:43.935186 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 298257us
I20250114 08:58:43.915198 22664 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:43.937705 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.942507 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.944330 21668 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.921128 22712 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:43.936702 22686 consensus_meta.cc:287] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.655s user 0.025s sys 0.005s
I20250114 08:58:43.914987 22583 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.944425 22684 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:43.917402 22681 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.967036 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:43.971318 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:43.974395 21668 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.952885 22670 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.980958 21703 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:43.985929 21668 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.970577 21550 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.936446 22667 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.958554 22674 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.969516 22713 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:43.981591 22694 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:43.991987 21668 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:43.860437 22723 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.992898 22696 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:44.019532 21551 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:43.957185 22686 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:43.981138 22670 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:58:44.012380 22708 consensus_meta.cc:287] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.569s user 0.000s sys 0.017s
I20250114 08:58:44.033973 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.038000 21554 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.010612 22694 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:44.019999 22712 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:44.060660 21627 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.061220 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.020913 22673 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:44.085417 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:43.972113 22702 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.036387 22670 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.093178 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.035444 22696 raft_consensus.cc:695] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:44.048835 22694 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.112764 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.052408 22712 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:44.112968 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.121268 22697 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.122848 21696 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.124061 22697 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:44.125411 21549 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.125298 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.086340 22685 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:44.081126 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 152484us
I20250114 08:58:44.128942 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:44.094050 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 354591us
W20250114 08:58:44.128562 22693 consensus_meta.cc:287] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.591s user 0.020s sys 0.000s
I20250114 08:58:44.150053 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:44.131709 22590 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.043258 22708 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.147660 22687 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.006s
I20250114 08:58:44.127732 22712 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.161185 22582 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.048722 22668 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.163036 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:44.167384 22582 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.168898 22241 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.170693 22590 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.172555 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.030975 22706 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.205811 21554 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.208204 21555 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:44.210788 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:44.213065 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.173677 21699 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:44.216446 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:44.216732 21553 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.152647 22671 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20250114 08:58:44.222421 22723 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.223573 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.223839 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.233263 21520 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.224399 22706 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:44.183826 22686 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.192970 22681 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.181193 22583 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.176827 22239 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:44.241953 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 83860.5us
I20250114 08:58:44.153417 22725 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20250114 08:58:44.244652 22667 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.233801 22664 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.217990 22645 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:44.248104 21520 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.250200 22590 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:44.256285 22717 consensus_meta.cc:287] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.547s user 0.011s sys 0.000s
W20250114 08:58:44.257151 22675 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.628s user 0.012s sys 0.004s
I20250114 08:58:44.258189 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:44.258872 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:44.259666 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:44.220912 22674 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.264571 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.266089 22730 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:44.266577 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:44.267439 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:44.245620 21667 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:44.276494 22717 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.287180 22717 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.289160 22668 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:44.290374 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 44879us
I20250114 08:58:44.290647 21552 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.233474 21546 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.205075 22693 consensus_queue.cc:1046] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20250114 08:58:44.276553 22700 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:44.281754 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.282938 21549 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.291740 22275 raft_consensus.cc:1235] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.294909 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:44.276062 22666 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.530s user 0.009s sys 0.006s
I20250114 08:58:44.277799 22686 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:44.302855 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.213229 22696 consensus_queue.cc:237] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.314680 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.316551 22720 consensus_queue.cc:1046] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250114 08:58:44.318101 22720 raft_consensus.cc:3049] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.318532 22720 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:44.319595 22720 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.325174 22720 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:44.340616 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 66689.9us
I20250114 08:58:44.300125 22726 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:44.292845 22645 raft_consensus.cc:695] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.273013 22670 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.343851 21520 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.350509 21520 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.279758 22675 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.355756 22675 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:44.301040 22702 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.360306 21704 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:44.306243 22699 consensus_meta.cc:287] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.655s user 0.006s sys 0.005s
I20250114 08:58:44.373976 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.306815 22727 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.369529 22708 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.297195 22730 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:44.375809 22703 consensus_meta.cc:287] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.783s user 0.019s sys 0.000s
I20250114 08:58:44.311446 22700 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.359697 22682 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:44.309556 22686 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.316067 22666 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.343389 22681 consensus_queue.cc:1046] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
W20250114 08:58:44.388378 21593 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.034s sys 0.164s
W20250114 08:58:44.374807 22715 consensus_meta.cc:287] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.654s user 0.012s sys 0.006s
I20250114 08:58:44.304435 22664 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.386502 22726 raft_consensus.cc:695] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:44.399008 22700 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.383105 22645 consensus_queue.cc:237] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.387255 22730 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.403779 22686 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.404946 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.394032 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.419924 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.398761 22668 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.423622 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:44.414978 21593 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.733s user 0.037s sys 0.172s
I20250114 08:58:44.429513 21703 raft_consensus.cc:2388] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:44.431000 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.431562 21630 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:44.399123 22703 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:44.433318 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 341870us
I20250114 08:58:44.436823 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.401759 22682 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.436235 21555 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:44.405053 22673 raft_consensus.cc:695] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:44.438606 21520 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.433456 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.434545 21669 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.452554 21553 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.344202 22712 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.451229 21698 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.453933 21669 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.454039 22693 consensus_queue.cc:1046] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:44.455036 22693 raft_consensus.cc:3049] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.455076 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.455507 22693 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.456786 22734 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.456424 22693 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.457487 22733 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.457516 22716 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.458626 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.459002 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.459465 21703 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:44.460080 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:44.460395 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.460536 21548 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:44.457755 22693 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.433573 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:44.461970 22583 consensus_queue.cc:1046] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:44.463033 22583 raft_consensus.cc:3049] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.463356 22583 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.404338 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.465665 21520 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.466435 21668 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:44.418491 22715 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.464159 22583 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.468530 21596 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:44.468670 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 69676.8us
I20250114 08:58:44.469007 22664 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.469116 22676 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.431375 22706 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.471717 22683 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:44.471740 22704 consensus_meta.cc:287] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.741s user 0.015s sys 0.007s
I20250114 08:58:44.472750 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:44.447547 22690 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.451730 22682 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:44.479857 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 91573.3us
I20250114 08:58:44.470902 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.480382 21555 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:44.469974 21547 consensus_meta.cc:287] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.744s user 0.011s sys 0.013s
I20250114 08:58:44.494645 22582 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.512216 22582 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:44.505601 21521 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.512300 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.512737 22582 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.513108 22582 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.513276 22731 consensus_queue.cc:1046] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20250114 08:58:44.513628 21596 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.513911 21552 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:44.492153 22684 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.508s user 0.002s sys 0.009s
I20250114 08:58:44.467921 22712 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:44.484288 22699 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:44.513010 21520 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.482896 22697 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.515530 21596 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.524837 21596 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.460117 22700 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.523572 21669 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.526878 21669 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.528824 21520 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.529700 21669 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.502970 22591 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.477388 22681 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20250114 08:58:44.530383 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.531780 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.463747 22726 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.544574 21669 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.538232 22706 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.519555 22725 consensus_queue.cc:1046] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20250114 08:58:44.560585 22699 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.560817 22716 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.539366 22583 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.561919 22721 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.562202 22681 raft_consensus.cc:3049] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:44.562330 22721 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.550032 22275 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.517189 22674 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.553587 22272 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.538631 22667 raft_consensus.cc:3049] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.514662 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.556550 22676 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250114 08:58:44.557164 21695 consensus_meta.cc:287] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.972s user 0.017s sys 0.000s
I20250114 08:58:44.517567 22670 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:44.561332 22699 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.565356 21520 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:44.519932 22679 consensus_meta.cc:287] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.507s user 0.012s sys 0.000s
I20250114 08:58:44.562657 22271 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.562642 22681 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.524327 22655 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.543354 22734 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.538156 22688 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.538338 22683 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.510836 22704 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.598958 21698 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:44.600153 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:44.553035 22697 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.522377 22731 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.566020 22728 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:44.576237 22667 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.605835 22712 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.563146 22591 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:44.624323 22722 consensus_meta.cc:287] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.655s user 0.011s sys 0.000s
I20250114 08:58:44.618197 22703 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:44.608673 22674 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250114 08:58:44.634616 22719 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.894s user 0.010s sys 0.002s
I20250114 08:58:44.554764 22673 consensus_queue.cc:237] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:44.615914 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 112746us
I20250114 08:58:44.636456 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:44.661237 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 136463us
I20250114 08:58:44.635107 22728 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.649415 22645 raft_consensus.cc:3049] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.644894 22723 raft_consensus.cc:3049] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.649106 22677 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.674090 21596 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.677793 21596 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.681500 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.682719 21596 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.682756 21623 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:44.623541 22721 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.667681 22728 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.665689 22715 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.669243 22645 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.644217 22740 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.132s
I20250114 08:58:44.688757 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:44.692843 22735 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20250114 08:58:44.674716 22723 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.667994 22687 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20250114 08:58:44.665093 22719 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.704403 21701 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:44.705420 22719 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:44.706743 22671 raft_consensus.cc:3049] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.707198 22671 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.698302 22582 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.696887 22700 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.661924 22681 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.677927 22677 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:44.728574 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 211804us
I20250114 08:58:44.755520 22744 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.729089 22687 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:44.724570 22742 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.698587 22270 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:44.681388 22239 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.726538 22741 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.700587 22686 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.754253 21700 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.717958 22582 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.708082 22671 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.753420 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1086) took 1126 ms. Trace:
I20250114 08:58:44.778801 21699 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.780824 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.787660 22722 raft_consensus.cc:3049] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.788276 22722 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.778946 21701 rpcz_store.cc:276] 0114 08:58:43.626900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:43.627050 (+ 150us) service_pool.cc:225] Handling call
0114 08:58:44.753042 (+1125992us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400371ed4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:44.753295 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2436192128}
I20250114 08:58:44.770845 22725 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.754670 22740 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:44.699378 22731 consensus_queue.cc:1159] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250114 08:58:44.803964 22582 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:44.705442 22730 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.799499 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.806372 22582 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:44.806869 22694 consensus_meta.cc:287] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.676s user 0.005s sys 0.005s
I20250114 08:58:44.772779 22687 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:44.789697 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 116463us
I20250114 08:58:44.811621 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:44.811929 22687 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.812709 22712 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:44.796640 22704 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.817016 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:44.817258 22582 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.816612 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.799970 22681 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:44.798233 22715 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.825748 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:44.807880 22677 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.799530 22591 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:44.816185 22671 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:44.818168 22684 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.834044 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:44.832254 22582 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.756801 22645 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.841334 21626 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:44.834381 22273 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:44.848613 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 159945us
I20250114 08:58:44.849607 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.851467 21596 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.831856 22704 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:44.853322 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 249390us
I20250114 08:58:44.818679 22694 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.835736 22684 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:44.870515 21555 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.852716 22591 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.852648 22727 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.879845 21596 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:44.816251 22723 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.908113 22744 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:44.875494 22715 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20250114 08:58:44.867605 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.914202 22275 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:44.914844 22742 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.934244 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:44.882530 22726 consensus_queue.cc:1046] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20250114 08:58:44.916119 22699 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.874894 22704 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.933215 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:44.917552 22684 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:44.919620 22708 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.937654 21700 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:44.866686 22725 consensus_queue.cc:1046] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250114 08:58:44.939463 21697 raft_consensus.cc:1235] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:44.940623 22272 raft_consensus.cc:2388] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:58:44.940764 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 396273us
I20250114 08:58:44.922246 22686 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.853791 22741 consensus_queue.cc:1046] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20250114 08:58:44.916234 22719 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 08:58:44.942777 22727 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.952982 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.919443 22583 consensus_queue.cc:1046] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20250114 08:58:44.915390 22745 consensus_queue.cc:1046] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20250114 08:58:44.966708 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.974529 22583 raft_consensus.cc:3049] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:44.975042 22583 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:44.910629 22722 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:44.978448 21669 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.980012 22726 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.962671 22271 raft_consensus.cc:1235] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:44.953152 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 98360.2us
I20250114 08:58:44.965344 22730 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:44.972517 21695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 909) took 1507 ms. Trace:
W20250114 08:58:44.994825 22693 consensus_meta.cc:287] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.534s user 0.010s sys 0.001s
I20250114 08:58:44.976651 22736 consensus_queue.cc:1046] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20250114 08:58:44.980046 22675 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:44.998643 21669 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:44.976112 22583 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.000003 22726 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.004145 22721 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.005569 22736 raft_consensus.cc:3049] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.006688 22272 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:45.007741 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.008639 22754 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:44.997830 22591 consensus_queue.cc:1046] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20250114 08:58:45.000538 22694 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:45.015218 22696 raft_consensus.cc:3049] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.017828 22679 raft_consensus.cc:3049] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:45.019742 22704 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:45.022944 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 142483us
I20250114 08:58:44.998123 21695 rpcz_store.cc:276] 0114 08:58:43.454333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:43.457408 (+ 3075us) service_pool.cc:225] Handling call
0114 08:58:43.519471 (+ 62063us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:43.526330 (+ 6859us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:58:44.604044 (+1077714us) log.cc:844] Serialized 14 byte log entry
0114 08:58:44.938267 (+334223us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:44.940880 (+ 2613us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:58:44.945578 (+ 4698us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:44.951943 (+ 6365us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:44.961806 (+ 9863us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7229,"raft.queue_time_us":3803,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5431,"thread_start_us":266903,"threads_started":1,"wal-append.queue_time_us":381276}
W20250114 08:58:45.040623 22682 consensus_meta.cc:287] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.521s user 0.010s sys 0.001s
I20250114 08:58:45.041064 22682 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.024700 21667 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:44.934422 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 959) took 1369 ms. Trace:
I20250114 08:58:45.030246 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.048043 21548 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.032011 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.049420 21669 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.050494 21669 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.032210 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:45.051652 22272 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:45.032931 21698 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.034376 22734 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.033335 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.043622 21596 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.046458 22682 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:45.030714 22736 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.050812 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.035868 22696 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.057205 22726 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:45.125707 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 48752.4us
W20250114 08:58:45.125854 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 48231.1us
I20250114 08:58:45.132831 21669 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.133946 22696 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.135124 22696 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:45.136657 22241 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.012s (SENT) [suppressed 4 similar messages]
I20250114 08:58:45.031663 22745 consensus_queue.cc:1046] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20250114 08:58:45.140547 22688 consensus_queue.cc:1046] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:45.141019 22734 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.141827 21669 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:45.141984 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 265297us
I20250114 08:58:45.042958 22583 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:45.133086 22739 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.413s user 0.004s sys 0.000s
I20250114 08:58:45.036134 22679 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:45.080118 21596 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:45.076910 22719 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.077822 22675 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.079056 22728 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:45.159895 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 79850.3us
I20250114 08:58:45.077765 22754 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:45.057984 22690 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.101s
I20250114 08:58:45.128623 22759 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.166028 21596 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:45.158638 22718 raft_consensus.cc:3049] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.173750 22718 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.158829 22688 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.162555 22681 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:45.053520 21547 rpcz_store.cc:276] 0114 08:58:43.561844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:43.562053 (+ 209us) service_pool.cc:225] Handling call
0114 08:58:43.562427 (+ 374us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:43.562638 (+ 211us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:58:44.520708 (+958070us) log.cc:844] Serialized 14 byte log entry
0114 08:58:44.880147 (+359439us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:44.884229 (+ 4082us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:58:44.916886 (+ 32657us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:44.920099 (+ 3213us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:44.931256 (+ 11157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9825,"raft.queue_time_us":20749,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6361,"thread_start_us":276236,"threads_started":1,"wal-append.queue_time_us":380540}
I20250114 08:58:45.176059 22740 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:45.186225 22740 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:45.186659 22740 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.148684 22655 consensus_queue.cc:1046] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 08:58:45.179265 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.180169 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
W20250114 08:58:45.193761 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 135871us
I20250114 08:58:45.198737 22590 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:45.199222 22590 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:45.187784 22725 raft_consensus.cc:3049] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.199915 22725 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:45.205983 21596 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.207644 21596 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.207749 21548 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:45.210899 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.171001 22685 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250114 08:58:45.214269 21552 raft_consensus.cc:1270] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:45.183884 22681 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:45.177793 22728 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:45.189620 22699 raft_consensus.cc:3049] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.188848 22754 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.205971 22590 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:58:45.219599 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 76474.8us
I20250114 08:58:45.211773 22687 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:45.214380 22745 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.217947 22759 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.218994 22681 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:45.254518 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 105175us
I20250114 08:58:45.200896 22725 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.222621 22699 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:45.219364 22760 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.257483 21669 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.244947 21547 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.220353 21596 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.221818 22741 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20250114 08:58:45.230444 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 93826.4us
I20250114 08:58:45.267086 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.255694 22723 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.277760 21552 raft_consensus.cc:2388] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:45.278808 21552 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:45.268196 21554 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.242467 22239 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:45.288285 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:45.284643 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 64403.3us
I20250114 08:58:45.287744 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.289606 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.290621 21596 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.294724 21628 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:45.296861 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.217803 22667 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.302100 21551 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.304754 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.246728 22687 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.313460 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.309070 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.277035 22747 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:45.316988 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.325184 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.317468 21628 raft_consensus.cc:1270] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:45.290386 22675 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.294533 22693 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.474040 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.285789 22745 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.252918 22754 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.301702 22725 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.309281 22731 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.313313 22727 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:45.339346 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.481169 21551 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:45.325083 22712 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:45.324749 21550 raft_consensus.cc:1235] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.257159 22718 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.363097 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:45.310765 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 84147.2us
I20250114 08:58:45.335603 22747 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.332587 22699 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.369236 22272 raft_consensus.cc:1235] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:45.371227 22671 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.534s user 0.010s sys 0.000s
I20250114 08:58:45.332852 22677 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:45.379824 22684 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.384240 22687 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:45.385660 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:58:45.395816 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 61395.5us
I20250114 08:58:45.498260 22275 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:58:45.399087 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.499464 21699 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.402778 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.501327 21700 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:45.411075 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.411859 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.412328 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.415154 22681 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.416854 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.508850 21698 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.408120 22591 consensus_queue.cc:1046] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250114 08:58:45.420665 22590 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.405496 21621 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:45.471047 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.472052 21553 raft_consensus.cc:1235] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.471909 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.476100 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 932) took 1487 ms. Trace:
I20250114 08:58:45.476835 21552 raft_consensus.cc:2388] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:45.475267 21626 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:45.515272 21554 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:45.515699 22590 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:45.515933 21546 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:45.484835 21551 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.520260 22747 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:45.513962 22718 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.505661 22677 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.526465 22677 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:45.526966 22677 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:45.489789 22712 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.527451 22677 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.529153 22712 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:45.533577 21551 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:45.436710 22740 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.514065 21697 rpcz_store.cc:276] 0114 08:58:43.989165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:43.995487 (+ 6322us) service_pool.cc:225] Handling call
0114 08:58:44.018841 (+ 23354us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:44.948950 (+930109us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:45.475786 (+526836us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b54001c0554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:45.476017 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2116679040}
I20250114 08:58:45.508054 21627 raft_consensus.cc:1270] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:45.509655 21521 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:45.489533 22727 raft_consensus.cc:695] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:45.521298 22591 raft_consensus.cc:3049] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.538373 21699 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
W20250114 08:58:45.491315 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 181426us
I20250114 08:58:45.541679 21624 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:45.546013 21696 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:45.558259 21596 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.559214 21699 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:45.550252 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.510428 22731 consensus_queue.cc:1046] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20250114 08:58:45.551285 22275 raft_consensus.cc:1235] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.554375 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.562448 22742 raft_consensus.cc:3049] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.563169 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:45.563266 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:45.567945 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.567356 21698 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:45.570242 22272 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:45.572005 21596 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.556661 22687 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:45.575001 21596 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.556317 22591 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:45.576498 21669 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:45.578169 22591 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.579644 22591 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.575901 22734 consensus_queue.cc:1046] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250114 08:58:45.581635 22677 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:45.583532 22697 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:45.584946 22697 raft_consensus.cc:3049] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.585314 22697 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.586200 22241 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:45.586124 22697 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.587428 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.587652 22697 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.588243 22730 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:45.592646 22730 raft_consensus.cc:695] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:45.592371 22727 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.593547 22730 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.595280 21702 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:45.600267 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:45.601301 22273 raft_consensus.cc:2388] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:58:45.589491 22772 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.589989 22712 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:45.604933 22712 raft_consensus.cc:695] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:45.605897 22712 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.608853 21552 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.591324 22716 consensus_queue.cc:1046] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:45.576236 22687 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:45.614471 22687 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.614744 22716 raft_consensus.cc:3049] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.615149 22716 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.574249 21547 raft_consensus.cc:1270] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:45.591727 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:45.575727 22745 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:45.597115 22767 consensus_queue.cc:1046] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:45.598294 21668 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.624117 22693 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.624279 22767 raft_consensus.cc:3049] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.624817 22767 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.625665 22675 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.569368 22696 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20250114 08:58:45.626024 22705 consensus_queue.cc:1046] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:45.626327 22675 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:45.627095 22705 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:45.572839 22708 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20250114 08:58:45.610352 22734 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:45.628947 22734 raft_consensus.cc:695] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.625690 22767 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.629935 22734 consensus_queue.cc:237] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.569943 22742 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:45.598440 22740 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:45.633759 21548 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:45.634629 22241 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:45.635485 22715 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:45.637081 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 196) took 2290 ms. Trace:
I20250114 08:58:45.611761 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.630343 22767 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.627270 22696 raft_consensus.cc:3049] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.641904 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.646895 22591 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.613381 21518 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.637459 21451 rpcz_store.cc:276] 0114 08:58:43.346632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:43.346800 (+ 168us) service_pool.cc:225] Handling call
0114 08:58:45.637022 (+2290222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:44.248971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:44.249673 (+ 702us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:44.249721 (+ 48us) write_op.cc:435] Acquired schema lock
0114 08:58:44.249743 (+ 22us) tablet.cc:659] Decoding operations
0114 08:58:44.255154 (+ 5411us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:44.255253 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:44.255291 (+ 38us) tablet.cc:682] Acquiring locks for 1 operations
0114 08:58:44.255622 (+ 331us) tablet.cc:698] Row locks acquired
0114 08:58:44.255644 (+ 22us) write_op.cc:260] PREPARE: finished
0114 08:58:44.255825 (+ 181us) write_op.cc:270] Start()
0114 08:58:44.255959 (+ 134us) write_op.cc:276] Timestamp: P: 1736845124255764 usec, L: 0
0114 08:58:44.256001 (+ 42us) op_driver.cc:348] REPLICATION: starting
0114 08:58:44.353796 (+ 97795us) log.cc:844] Serialized 602 byte log entry
0114 08:58:44.807765 (+453969us) op_driver.cc:464] REPLICATION: finished
0114 08:58:45.041522 (+233757us) write_op.cc:301] APPLY: starting
0114 08:58:45.041620 (+ 98us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:45.041741 (+ 121us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:45.041889 (+ 148us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:45.045546 (+ 3657us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:45.045767 (+ 221us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 08:58:45.045839 (+ 72us) write_op.cc:312] APPLY: finished
0114 08:58:45.046141 (+ 302us) log.cc:844] Serialized 22 byte log entry
0114 08:58:45.046328 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:45.046419 (+ 91us) write_op.cc:454] Released schema lock
0114 08:58:45.046475 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:45.046880 (+ 405us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:45.046914 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":233439,"apply.run_cpu_time_us":2182,"apply.run_wall_time_us":5339,"num_ops":1,"prepare.queue_time_us":405438,"prepare.run_cpu_time_us":29172,"prepare.run_wall_time_us":300491,"raft.queue_time_us":271207,"raft.run_cpu_time_us":56372,"raft.run_wall_time_us":223332,"replication_time_us":551581,"thread_start_us":757500,"threads_started":4,"wal-append.queue_time_us":152682}]]}
I20250114 08:58:45.654143 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.695937 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:45.697578 22272 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:58:45.768339 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.771242 22684 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:58:45.771903 22684 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:45.772994 22684 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.778103 22645 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.656105 22696 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.660621 21453 catalog_manager.cc:5526] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 2, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:45.656702 22740 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.786423 21703 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:45.631837 22742 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.616101 22716 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.697436 22715 raft_consensus.cc:695] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:45.801919 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.806597 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:45.817797 22679 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.819937 22679 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:45.821593 21596 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.822417 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.823228 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:45.824594 22772 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.824617 22591 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:45.825865 22722 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.781628 22759 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 08:58:45.857932 22667 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.846159 22681 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:45.883648 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:45.882512 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.889640 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.859160 22742 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.845431 22755 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:45.867318 22715 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.899768 22703 raft_consensus.cc:3049] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:45.900343 22703 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.901429 22703 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.906347 22703 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:45.868080 22655 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250114 08:58:45.881116 22687 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.918198 22772 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:45.918782 22772 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:45.927758 21630 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:45.928678 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:45.832752 22696 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:45.916926 22786 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:45.958659 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:45.954748 22786 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:45.968709 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.973245 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:45.976431 21630 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:45.990852 22704 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:45.992695 21669 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:45.993192 21630 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:45.995157 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:45.997216 21669 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
W20250114 08:58:46.001716 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 350318us
I20250114 08:58:46.004714 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.007000 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:46.008682 21700 raft_consensus.cc:1235] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:46.009168 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:46.010196 21704 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:58:45.995640 22785 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:46.011735 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:45.997761 22696 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.012914 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:46.021364 21630 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:46.021485 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:46.018282 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 251757us
I20250114 08:58:46.014688 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:46.024349 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:46.019994 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:46.018394 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.025599 21698 raft_consensus.cc:2388] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:58:46.010253 22767 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:46.027174 21596 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.006129 22740 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:46.029245 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:46.029536 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:46.040542 21626 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:46.039317 21627 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:46.037281 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:46.043670 22272 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:58:46.013867 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:46.047304 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:46.048705 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet 3b369b35736c413ca069db061ed36bc5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d7732324fb5f48d3b16a29d4bb2584ed is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:46.042635 21628 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:46.050060 22699 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.056509 21624 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:46.058449 21627 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:46.068609 21629 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:46.026674 22687 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.035650 22745 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.071887 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:46.084308 21518 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.043706 22767 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:46.082505 22783 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20250114 08:58:46.038363 22740 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:46.128876 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 112377us
I20250114 08:58:46.072888 21547 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:46.131111 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:46.116565 21622 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:46.132027 22272 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:46.034683 22786 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.098706 22772 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.099619 22682 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.146284 21698 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:46.146937 21553 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:46.153354 21520 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.155572 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.156399 22682 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.156721 21520 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.158166 21520 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.158528 22682 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.159160 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.145201 21593 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:46.145596 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:46.160109 21625 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:46.160120 22693 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:46.155704 22582 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.160765 22693 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:46.138818 21669 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:46.161180 22693 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:46.159201 22682 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.140355 21703 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:46.164317 21520 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.150449 21452 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:46.161273 21555 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250114 08:58:46.137763 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 93938us
I20250114 08:58:46.159337 21554 raft_consensus.cc:1270] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:46.165362 21452 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 2, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 2 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:46.185822 21520 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.184943 22679 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:46.191975 22767 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.194725 21596 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:46.195456 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 56720.4us
I20250114 08:58:46.196332 21596 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.197649 21596 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.200317 22760 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.203266 22736 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.204352 22736 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.206476 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.206133 22703 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:46.209834 22690 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.209323 22590 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:46.209494 22679 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:46.210678 22690 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.211850 22679 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.208853 22755 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.213415 22690 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.213685 22740 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.184405 22704 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.223692 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.229261 22690 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.232987 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.205513 22723 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.216861 22703 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:46.254472 22703 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.238572 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.240180 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:46.240775 21695 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:46.241314 22590 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.241394 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.241583 22727 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:46.241401 22712 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:46.244902 21704 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:46.245904 21623 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:46.246939 22685 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:58:46.218161 22715 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20250114 08:58:46.247876 22772 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.268313 22685 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:46.216167 22786 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:46.252287 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:46.263147 21520 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.270782 21521 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:46.273679 21549 raft_consensus.cc:2388] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:46.250658 22716 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.275672 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.266830 22761 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:46.277410 21520 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.278333 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 971) took 1005 ms. Trace:
I20250114 08:58:46.278903 21520 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.278594 21549 rpcz_store.cc:276] 0114 08:58:45.269070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:45.269196 (+ 126us) service_pool.cc:225] Handling call
0114 08:58:46.252079 (+982883us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b5800000f98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:58:46.274686 (+ 22607us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7287680}
I20250114 08:58:46.281787 22682 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.276840 22590 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:46.282300 22715 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:46.282792 22590 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:46.283237 22590 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:46.283294 22685 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.277998 22677 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.284665 22727 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:46.286247 22690 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.277313 22772 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.294719 22582 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250114 08:58:46.284726 22685 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:46.280093 22704 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.278071 22761 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:46.306711 22760 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.293890 22682 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:46.327836 22708 raft_consensus.cc:3049] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:46.330019 22712 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:46.330691 21550 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:46.309599 22761 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.316625 22715 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.338873 22704 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:46.338757 22708 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:46.357043 22715 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:46.373646 22803 log.cc:926] Time spent T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20250114 08:58:46.371165 22704 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.380656 22704 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.380807 22715 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:46.392151 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:46.393020 21623 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:46.393797 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:46.394544 21550 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:46.395956 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:46.396775 22272 raft_consensus.cc:2388] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:58:46.396971 22777 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.543s user 0.004s sys 0.000s
I20250114 08:58:46.397125 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:46.398247 22715 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:46.398718 22715 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:46.399181 22715 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:46.404310 22727 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.405146 22693 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.359706 22760 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.343187 22682 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.415711 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:46.416014 22693 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.405165 22727 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.418179 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:46.423192 22272 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:46.425621 21550 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 5
I20250114 08:58:46.431069 21701 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
W20250114 08:58:46.425264 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 79562.8us
I20250114 08:58:46.420681 22760 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.433159 22712 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:46.451277 22690 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.462031 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:46.470602 21520 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.426419 22708 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.493829 22272 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:46.499130 21520 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.518126 22809 consensus_queue.cc:1046] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250114 08:58:46.522603 21593 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:46.497381 21695 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:46.534466 22693 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:58:46.537407 22693 raft_consensus.cc:695] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:46.471443 22690 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.480235 22808 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:46.517513 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:46.471980 22712 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.504187 22590 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.542577 22704 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.573895 22716 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.617084 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:46.617972 21552 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:46.619612 21594 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:46.610502 22682 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.612929 22715 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.628545 22682 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.632092 22760 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.611933 22655 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:46.616204 22693 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.623595 22667 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:46.638370 22655 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.643165 22590 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.645499 21550 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:46.661814 22728 raft_consensus.cc:2758] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250114 08:58:46.665172 21630 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:46.667425 22783 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:46.670760 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:46.672777 22779 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.672899 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.673652 22779 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.630389 22786 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:46.615708 22716 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.676136 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 956) took 1389 ms. Trace:
I20250114 08:58:46.659986 22667 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:58:46.647925 22806 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20250114 08:58:46.676779 22687 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.793112 22687 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.677734 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.797775 21695 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:46.798828 22687 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.680014 22655 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:46.680251 22772 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.682490 22712 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.801618 22712 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:46.802201 22712 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:46.683302 22786 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.806645 22786 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:46.684180 22690 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.692790 21622 rpcz_store.cc:276] 0114 08:58:45.286674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:45.286796 (+ 122us) service_pool.cc:225] Handling call
0114 08:58:46.675868 (+1389072us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b5400152cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:46.676016 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1767904000}
I20250114 08:58:46.702044 22667 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:46.810063 22712 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.810844 22712 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:58:46.676434 22779 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.704425 21700 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:46.705747 21594 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:46.706549 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:46.817481 21555 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
W20250114 08:58:46.819391 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 113730us
I20250114 08:58:46.820653 22772 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:46.821385 22772 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.826083 22671 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:46.827641 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:46.723883 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 933) took 1428 ms (client timeout 1654 ms). Trace:
I20250114 08:58:46.828819 21623 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:46.724862 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:46.832737 21551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 35) took 1262 ms (client timeout 1578 ms). Trace:
I20250114 08:58:46.832953 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:46.833205 21551 rpcz_store.cc:269] 0114 08:58:45.570326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.724431 (+1154105us) service_pool.cc:225] Handling call
0114 08:58:46.832664 (+108233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:46.833863 21630 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:46.728144 21702 raft_consensus.cc:1270] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:46.835945 22671 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.836567 22671 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:46.829321 21701 rpcz_store.cc:269] 0114 08:58:45.295886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:45.296063 (+ 177us) service_pool.cc:225] Handling call
0114 08:58:46.723610 (+1427547us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540018a6d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:46.723791 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2311264128}
I20250114 08:58:46.839592 22671 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.843716 21594 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:46.850281 22723 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:46.850888 22723 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:46.851416 22723 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:46.864521 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:46.865592 21551 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:46.749495 21453 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55888 (request call id 194) took 1805 ms. Trace:
I20250114 08:58:46.773252 22742 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:46.871584 22742 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.870738 22667 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.790539 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:46.875758 22272 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:46.755993 21452 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 69) took 1716 ms. Trace:
I20250114 08:58:46.778043 22708 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:46.719250 22761 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.882582 22742 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.883427 22742 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.774160 21451 catalog_manager.cc:5526] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to <none>, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:46.868836 21453 rpcz_store.cc:276] 0114 08:58:44.944523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:44.944681 (+ 158us) service_pool.cc:225] Handling call
0114 08:58:45.626103 (+681422us) spinlock_profiling.cc:243] Waited 78.5 ms on lock 0x7b54001be754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f552d50 00007fa63f5d5f2a 00007fa6452f1dec 00007fa6453e9dee 00007fa64393cfc4 00007fa64393cf52 00007fa64393ce81 00007fa64393cdfc 00007fa64393c0b2 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:46.749367 (+1123264us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:45.986916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:45.987529 (+ 613us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:45.987570 (+ 41us) write_op.cc:435] Acquired schema lock
0114 08:58:45.987591 (+ 21us) tablet.cc:659] Decoding operations
0114 08:58:45.987953 (+ 362us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:45.988026 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:45.988063 (+ 37us) tablet.cc:682] Acquiring locks for 1 operations
0114 08:58:45.988264 (+ 201us) tablet.cc:698] Row locks acquired
0114 08:58:45.988285 (+ 21us) write_op.cc:260] PREPARE: finished
0114 08:58:45.988431 (+ 146us) write_op.cc:270] Start()
0114 08:58:45.988572 (+ 141us) write_op.cc:276] Timestamp: P: 1736845125988385 usec, L: 0
0114 08:58:45.988594 (+ 22us) op_driver.cc:348] REPLICATION: starting
0114 08:58:45.989237 (+ 643us) log.cc:844] Serialized 543 byte log entry
0114 08:58:46.143136 (+153899us) op_driver.cc:464] REPLICATION: finished
0114 08:58:46.303681 (+160545us) write_op.cc:301] APPLY: starting
0114 08:58:46.303764 (+ 83us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:46.303851 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:46.306295 (+ 2444us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:46.306579 (+ 284us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:46.306747 (+ 168us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 08:58:46.306797 (+ 50us) write_op.cc:312] APPLY: finished
0114 08:58:46.306994 (+ 197us) log.cc:844] Serialized 22 byte log entry
0114 08:58:46.307189 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:46.307262 (+ 73us) write_op.cc:454] Released schema lock
0114 08:58:46.307300 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:46.307635 (+ 335us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:46.307670 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11777,"reported_tablets":1,"spinlock_wait_cycles":180597632,"child_traces":[["op",{"apply.queue_time_us":160298,"apply.run_cpu_time_us":1941,"apply.run_wall_time_us":3948,"num_ops":1,"prepare.queue_time_us":187541,"prepare.run_cpu_time_us":2613,"prepare.run_wall_time_us":2613,"replication_time_us":154397,"thread_start_us":328651,"threads_started":2}]]}
I20250114 08:58:46.884423 22761 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.887440 22772 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.885828 21451 catalog_manager.cc:5526] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 2 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:46.884023 22742 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:46.687201 22801 log.cc:926] Time spent T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 08:58:46.890682 22742 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.877183 21452 rpcz_store.cc:276] 0114 08:58:45.039166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:45.045408 (+ 6242us) service_pool.cc:225] Handling call
0114 08:58:46.755904 (+1710496us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:46.214460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:46.216877 (+ 2417us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:46.220796 (+ 3919us) write_op.cc:435] Acquired schema lock
0114 08:58:46.225093 (+ 4297us) tablet.cc:659] Decoding operations
0114 08:58:46.267689 (+ 42596us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:46.267741 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:46.267767 (+ 26us) tablet.cc:682] Acquiring locks for 2 operations
0114 08:58:46.268049 (+ 282us) tablet.cc:698] Row locks acquired
0114 08:58:46.268062 (+ 13us) write_op.cc:260] PREPARE: finished
0114 08:58:46.268216 (+ 154us) write_op.cc:270] Start()
0114 08:58:46.268330 (+ 114us) write_op.cc:276] Timestamp: P: 1736845126268170 usec, L: 0
0114 08:58:46.268348 (+ 18us) op_driver.cc:348] REPLICATION: starting
0114 08:58:46.308549 (+ 40201us) log.cc:844] Serialized 988 byte log entry
0114 08:58:46.313841 (+ 5292us) op_driver.cc:464] REPLICATION: finished
0114 08:58:46.314334 (+ 493us) write_op.cc:301] APPLY: starting
0114 08:58:46.314419 (+ 85us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:46.314550 (+ 131us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:46.314677 (+ 127us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:46.315024 (+ 347us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:46.315271 (+ 247us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 08:58:46.315318 (+ 47us) write_op.cc:312] APPLY: finished
0114 08:58:46.316397 (+ 1079us) log.cc:844] Serialized 28 byte log entry
0114 08:58:46.316591 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:46.324943 (+ 8352us) write_op.cc:454] Released schema lock
0114 08:58:46.324993 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:46.714273 (+389280us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b54001be754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5c9198 00007fa642fe03b1 00007fa642fdfb88 00007fa642fe4821 00007fa642fe47d9 00007fa642fe4769 00007fa642fe4731 00007fa642fe395d 00007fa6456d1234 00007fa6456d1069
0114 08:58:46.714522 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:46.714562 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":648539,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":273,"apply.run_cpu_time_us":1954,"apply.run_wall_time_us":13089,"num_ops":2,"prepare.queue_time_us":2043,"prepare.run_cpu_time_us":6731,"prepare.run_wall_time_us":500005,"replication_time_us":45356,"spinlock_wait_cycles":10588032}]]}
I20250114 08:58:46.745349 22716 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:46.767647 22715 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:46.896261 22685 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.586s user 0.009s sys 0.002s
I20250114 08:58:46.908497 22667 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.910485 22723 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:46.925654 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:46.929819 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:46.930694 21552 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:46.933226 21555 raft_consensus.cc:2388] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:58:46.934863 21553 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.028s sys 0.099s
W20250114 08:58:46.934082 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 228320us
W20250114 08:58:46.938402 21553 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.028s sys 0.099s
I20250114 08:58:46.940160 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:46.941033 21701 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:47.461928 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:46.994573 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:47.511065 21630 raft_consensus.cc:2388] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:58:47.461045 21701 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.519s user 0.018s sys 0.000s
I20250114 08:58:47.513437 21701 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:47.012989 22239 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:47.515053 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 502159us
W20250114 08:58:47.463575 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 523804us
W20250114 08:58:47.719242 22827 log.cc:926] Time spent T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250114 08:58:47.805532 22821 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.240348 22851 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:47.046372 21552 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:48.240499 22843 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:48.241005 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 995) took 1420 ms. Trace:
I20250114 08:58:48.055821 22831 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:58:48.001999 21520 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.420s user 0.037s sys 0.047s
I20250114 08:58:48.241679 22831 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.242046 22843 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.241520 21622 rpcz_store.cc:276] 0114 08:58:46.820628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.820779 (+ 151us) service_pool.cc:225] Handling call
0114 08:58:46.821207 (+ 428us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:46.821619 (+ 412us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:46.821659 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:46.822232 (+ 573us) log.cc:844] Serialized 26 byte log entry
0114 08:58:47.462869 (+640637us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:47.462927 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:47.463021 (+ 94us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:48.240756 (+777735us) raft_consensus.cc:1680] finished
0114 08:58:48.240774 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:48.240926 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5463,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":149,"thread_start_us":324874,"threads_started":1,"wal-append.queue_time_us":710807}
I20250114 08:58:48.242815 22843 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:47.791757 21668 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.088s user 0.014s sys 0.007s
I20250114 08:58:47.600350 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:47.705196 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:47.597316 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:48.297808 21699 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:47.646556 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:48.299542 21701 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:47.599475 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:48.061015 22832 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.304440 21703 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:48.304601 22832 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.307469 22832 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.307978 22832 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:58:47.812855 22237 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.275s user 0.052s sys 0.120s
W20250114 08:58:47.035385 22240 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.065s sys 0.122s
W20250114 08:58:48.310886 22240 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.899s user 0.069s sys 0.141s
I20250114 08:58:48.311573 21701 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
W20250114 08:58:47.499025 22241 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.001s sys 0.012s
W20250114 08:58:48.314707 22241 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.705s user 0.001s sys 0.013s
W20250114 08:58:48.315095 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 1.7055e+06us
W20250114 08:58:48.246997 21668 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.543s user 0.015s sys 0.007s
W20250114 08:58:48.317514 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 1.61399e+06us
I20250114 08:58:46.934370 22667 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.321389 22667 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.323707 22667 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:47.025151 22671 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.326608 22671 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:47.690105 22673 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.329466 22673 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.331668 22673 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:47.668742 22685 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.332722 22858 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.333040 22685 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: NON_VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: true } }
I20250114 08:58:48.333370 22858 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.335381 22685 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:47.025079 22690 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.336737 22690 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.338894 22690 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:47.824919 22693 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.107s user 0.009s sys 0.014s
W20250114 08:58:48.340858 22693 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.623s user 0.011s sys 0.014s
I20250114 08:58:48.341466 22693 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.342003 22693 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.344358 21697 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:47.594774 22727 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:47.482586 22697 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:47.674201 22734 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.349197 22697 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.349908 22734 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.350602 22857 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.351215 22857 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.352020 22734 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:47.474778 22742 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.353322 22857 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:47.161382 22772 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:47.528789 22779 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:47.522406 22715 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:47.845646 22785 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.809s user 0.016s sys 0.035s
I20250114 08:58:47.581729 22766 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.003786 22786 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:47.695338 22730 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.059s user 0.012s sys 0.006s
I20250114 08:58:47.719986 22794 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:47.801844 22796 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:58:47.828506 22808 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.213s user 0.013s sys 0.010s
W20250114 08:58:47.722133 22810 log.cc:926] Time spent T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.239s user 0.001s sys 0.000s
W20250114 08:58:48.048501 22812 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.514s user 0.003s sys 0.014s
W20250114 08:58:47.721246 22813 log.cc:926] Time spent T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20250114 08:58:48.015705 22815 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.236s user 0.000s sys 0.009s
W20250114 08:58:47.187860 22809 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.004s sys 0.054s
I20250114 08:58:47.788976 22814 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:48.070353 22833 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.072350 22834 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.076200 22835 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.077528 22836 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.129691 22824 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.130275 22837 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.973968 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.153172 22838 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:46.963706 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:48.175832 22839 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.185071 22840 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.186707 22841 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.187904 22842 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.206801 22844 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.208060 22845 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:47.025372 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.217202 22847 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.225100 22846 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.226508 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.226928 22848 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.227761 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:48.228940 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.229283 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.235248 22850 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.235644 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1174) took 2000 ms. Trace:
I20250114 08:58:48.239948 22821 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.241050 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1002) took 1311 ms. Trace:
I20250114 08:58:48.047627 22829 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:48.242303 21520 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.660s user 0.039s sys 0.048s
I20250114 08:58:47.597621 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:48.245729 22843 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:47.777736 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:48.248312 21696 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:48.253470 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.243811 22831 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.701512 21695 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:48.702059 21695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 949) took 1106 ms. Trace:
I20250114 08:58:48.702419 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.702687 22831 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.703176 21548 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:48.703261 22831 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.261902 21521 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:48.704910 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 443073us
I20250114 08:58:48.265623 22854 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.240888 22851 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.269867 22855 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:48.270829 21595 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.000s user 0.015s sys 0.032s
I20250114 08:58:48.273020 21700 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:58:48.274546 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:48.277654 22856 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:48.709506 21595 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.438s user 0.016s sys 0.033s
I20250114 08:58:48.709862 22856 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.276940 21452 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 2 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:48.711529 21452 catalog_manager.cc:5526] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 2 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:48.712441 22856 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.712743 22870 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.290789 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 197) took 1858 ms. Trace:
W20250114 08:58:48.309731 22237 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.771s user 0.053s sys 0.121s
I20250114 08:58:48.713263 21452 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 2, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:48.714679 21452 catalog_manager.cc:5526] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 2, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 2 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:48.308429 22832 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.716161 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.716379 21452 catalog_manager.cc:5526] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 2 to 1, leader changed from <none> to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 1 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:48.718777 22832 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.718231 21452 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 2 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:48.719504 22832 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.721206 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.722216 22268 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:48.713485 21451 rpcz_store.cc:276] 0114 08:58:46.432138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.432288 (+ 150us) service_pool.cc:225] Handling call
0114 08:58:48.290593 (+1858305us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:46.891220 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:46.891707 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:46.891744 (+ 37us) write_op.cc:435] Acquired schema lock
0114 08:58:46.891760 (+ 16us) tablet.cc:659] Decoding operations
0114 08:58:46.892140 (+ 380us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:46.892198 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:46.892227 (+ 29us) tablet.cc:682] Acquiring locks for 2 operations
0114 08:58:46.897292 (+ 5065us) tablet.cc:698] Row locks acquired
0114 08:58:46.897322 (+ 30us) write_op.cc:260] PREPARE: finished
0114 08:58:46.897501 (+ 179us) write_op.cc:270] Start()
0114 08:58:46.897645 (+ 144us) write_op.cc:276] Timestamp: P: 1736845126897450 usec, L: 0
0114 08:58:46.897675 (+ 30us) op_driver.cc:348] REPLICATION: starting
0114 08:58:46.898427 (+ 752us) log.cc:844] Serialized 959 byte log entry
0114 08:58:47.755340 (+856913us) op_driver.cc:464] REPLICATION: finished
0114 08:58:48.037378 (+282038us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63c67681e 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa642fe0d20
0114 08:58:48.049332 (+ 11954us) write_op.cc:301] APPLY: starting
0114 08:58:48.049428 (+ 96us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:48.049573 (+ 145us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:48.049596 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:48.050031 (+ 435us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:48.050213 (+ 182us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 08:58:48.050265 (+ 52us) write_op.cc:312] APPLY: finished
0114 08:58:48.050483 (+ 218us) log.cc:844] Serialized 28 byte log entry
0114 08:58:48.051312 (+ 829us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:48.051405 (+ 93us) write_op.cc:454] Released schema lock
0114 08:58:48.051460 (+ 55us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":261746,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":293669,"apply.run_cpu_time_us":1799,"apply.run_wall_time_us":2465,"num_ops":2,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":2833,"prepare.run_wall_time_us":7469,"replication_time_us":857446,"spinlock_wait_cycles":4947712,"thread_start_us":301547,"threads_started":1}]]}
I20250114 08:58:48.349292 22858 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.724329 22858 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.724990 22858 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.727555 22858 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:46.934880 22723 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.730005 22723 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.353291 22742 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:47.534273 22768 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.353879 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.755362 22866 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.755703 22273 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:48.755568 22768 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.354481 22697 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:48.756210 22866 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.757817 22697 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.758217 22768 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:48.758318 22697 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.759217 22866 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.359479 22734 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.361757 22693 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.324262 22667 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.364809 22857 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.760118 22734 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.760378 22667 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.597150 21550 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:48.760076 22697 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:48.761528 21550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1000) took 1833 ms (client timeout 1764 ms). Trace:
W20250114 08:58:48.761896 21550 rpcz_store.cc:269] 0114 08:58:46.927754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.932970 (+ 5216us) service_pool.cc:225] Handling call
0114 08:58:48.761457 (+1828487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:48.762401 22734 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:48.762661 22667 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.762951 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:48.597380 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 2.07982e+06us
I20250114 08:58:48.597738 21555 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:48.368139 22275 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:58:48.599912 21554 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:48.765121 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.765390 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.595567 21698 rpcz_store.cc:276] 0114 08:58:46.234993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.235183 (+ 190us) service_pool.cc:225] Handling call
0114 08:58:46.257264 (+ 22081us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:47.493163 (+1235899us) raft_consensus.cc:1529] Early marking committed up to index 5
0114 08:58:47.493207 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:47.493263 (+ 56us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:58:47.493305 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:48.235433 (+742128us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b540023dcd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:48.235480 (+ 47us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:48.235596 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":727898,"raft.queue_time_us":596720,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":1965,"spinlock_wait_cycles":1471093120}
I20250114 08:58:48.765920 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.766322 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.766548 21621 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:48.766927 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:48.767261 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.767906 21627 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:48.599157 21552 rpcz_store.cc:276] 0114 08:58:46.929327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.929485 (+ 158us) service_pool.cc:225] Handling call
0114 08:58:48.240970 (+1311485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:48.367925 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:58:48.602826 22809 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.082s user 0.020s sys 0.086s
W20250114 08:58:48.600986 22808 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.983s user 0.020s sys 0.010s
I20250114 08:58:48.557075 22673 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:58:48.602316 22730 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.961s user 0.015s sys 0.011s
I20250114 08:58:48.564417 22727 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:48.669334 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 1.0717e+06us
I20250114 08:58:48.787461 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:48.788723 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:48.601558 22785 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.567s user 0.020s sys 0.044s
I20250114 08:58:48.415900 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.393227 22817 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.668370 22820 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.666340 22861 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.601689 22839 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:48.848590 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet 3b369b35736c413ca069db061ed36bc5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d7732324fb5f48d3b16a29d4bb2584ed is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:48.392007 22690 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.365020 22685 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.600787 22843 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.374679 22671 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:48.604236 21546 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:48.338913 22862 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.682869 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 34) took 2393 ms. Trace:
I20250114 08:58:48.686458 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 48) took 2379 ms. Trace:
I20250114 08:58:48.602278 22786 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.600904 22829 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.602470 22772 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.602355 22715 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.602422 22794 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.690048 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 41) took 2270 ms. Trace:
I20250114 08:58:48.602531 22848 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.737753 22854 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.875813 22848 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.604631 22850 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.681480 22868 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:48.604178 22835 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.601876 22838 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.601528 22841 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.601548 22834 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.601467 22836 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.696733 22863 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.699393 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.602700 22845 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.700341 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.704183 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.601457 22840 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.602456 22846 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.709134 21696 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:48.713035 22856 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.292515 22853 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.734848 22742 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.736032 22870 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.720131 22832 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.728168 22858 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.601437 22842 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.730644 22723 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:48.602403 22766 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.604331 22796 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.602113 22824 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.602468 22844 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.739457 22818 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:48.601385 22833 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.747984 21622 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250114 08:58:48.702389 21695 rpcz_store.cc:276] 0114 08:58:47.595853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:47.638604 (+ 42751us) service_pool.cc:225] Handling call
0114 08:58:48.702003 (+1063399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:48.601187 22847 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.601979 22837 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.751005 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.752131 22872 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.603698 22814 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.760493 22857 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.596622 21549 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:48.767501 21624 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:48.360146 22823 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:48.764163 21667 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:48.770821 22874 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.773350 22871 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:48.763943 22831 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.776734 22864 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.768827 22712 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:48.841343 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 152528us
I20250114 08:58:48.842573 22866 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.844321 22875 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.835381 22274 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
W20250114 08:58:48.845357 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 81414.8us
W20250114 08:58:48.846629 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 140895us
I20250114 08:58:48.848203 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.776409 22734 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.842268 22667 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.776232 22808 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:48.860497 21550 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:48.839330 22785 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:48.918900 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.895215 22267 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:48.860296 21625 rpcz_store.cc:276] 0114 08:58:46.304402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.304591 (+ 189us) service_pool.cc:225] Handling call
0114 08:58:46.331422 (+ 26831us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:46.331765 (+ 343us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:46.331791 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:46.332200 (+ 409us) log.cc:844] Serialized 26 byte log entry
0114 08:58:46.522245 (+190045us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:46.522306 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:46.522405 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:48.684013 (+2161608us) raft_consensus.cc:1680] finished
0114 08:58:48.684038 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:48.684178 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":884,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":2713,"thread_start_us":189624,"threads_started":1,"wal-append.queue_time_us":1161984}
I20250114 08:58:48.859803 21704 rpcz_store.cc:276] 0114 08:58:46.286827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.287348 (+ 521us) service_pool.cc:225] Handling call
0114 08:58:46.287699 (+ 351us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:46.288042 (+ 343us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:46.288071 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:46.292992 (+ 4921us) log.cc:844] Serialized 26 byte log entry
0114 08:58:46.476925 (+183933us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:46.476979 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:46.477071 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:48.047811 (+1570740us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:58:48.680225 (+632414us) raft_consensus.cc:1680] finished
0114 08:58:48.680244 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:48.680453 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4964,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":5484,"spinlock_wait_cycles":12436352,"thread_start_us":180930,"threads_started":1,"wal-append.queue_time_us":1162872}
I20250114 08:58:48.939316 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:48.926389 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 162306us
I20250114 08:58:48.939940 21622 raft_consensus.cc:2363] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:48.860113 21593 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:48.879671 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 111645us
I20250114 08:58:48.940639 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.870977 21553 rpcz_store.cc:276] 0114 08:58:46.417777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.418665 (+ 888us) service_pool.cc:225] Handling call
0114 08:58:46.418975 (+ 310us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:46.419289 (+ 314us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:46.419315 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:46.419730 (+ 415us) log.cc:844] Serialized 26 byte log entry
0114 08:58:46.940370 (+520640us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:46.940428 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:46.940543 (+ 115us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:48.015075 (+1074532us) spinlock_profiling.cc:243] Waited 915 us on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:58:48.685441 (+670366us) raft_consensus.cc:1680] finished
0114 08:58:48.685456 (+ 15us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:48.688390 (+ 2934us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":916,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":4905,"spinlock_wait_cycles":2104576,"thread_start_us":520252,"threads_started":1,"wal-append.queue_time_us":1254861}
I20250114 08:58:48.866459 22821 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.893255 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 953) took 1296 ms. Trace:
I20250114 08:58:48.847177 22727 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.836287 22673 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:48.858980 22861 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.749519 22855 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:48.894449 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 122035us
I20250114 08:58:48.913918 22840 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.859689 22690 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.917913 22839 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.859833 22685 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.859892 22843 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.859566 22862 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.993769 21553 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:48.994333 22685 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:48.928025 21630 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:48.994560 21548 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:48.849804 22817 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.997017 22824 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.901885 22863 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.998309 22817 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.934892 21627 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:48.840305 22697 raft_consensus.cc:2758] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250114 08:58:48.945998 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:48.946130 22876 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:49.023015 21551 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:58:48.951679 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 95757.6us
I20250114 08:58:48.961163 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.024879 21629 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:48.951200 21624 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:48.962829 21594 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:48.949118 21696 rpcz_store.cc:276] 0114 08:58:47.597055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:47.600078 (+ 3023us) service_pool.cc:225] Handling call
0114 08:58:48.893194 (+1293116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:48.965536 22831 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:49.026155 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 63422.3us
I20250114 08:58:48.973049 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:48.907011 22704 raft_consensus.cc:3049] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:48.870872 22820 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.919340 22846 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:48.993923 22844 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.917243 22836 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.915289 22723 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:48.921178 22845 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:48.978526 22886 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:48.937361 22742 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:48.914636 22841 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.906329 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:48.904824 21547 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:58:48.921882 22851 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.897794 22856 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.932835 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:48.910342 22842 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.020668 21628 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
W20250114 08:58:49.021070 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 44334.8us
I20250114 08:58:48.965871 22881 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:48.976821 22868 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:48.971704 22848 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.028414 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.904910 22858 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.942638 22667 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.875876 22854 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.032967 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.031077 21625 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.034406 21594 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:48.942788 22734 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:49.036087 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 116941us
I20250114 08:58:49.035884 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.037372 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.072286 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.899636 22853 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.073454 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:48.966842 22871 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:48.944875 22866 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:49.042167 21551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 998) took 2113 ms (client timeout 2042 ms). Trace:
I20250114 08:58:49.029618 22693 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:49.030076 22831 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.048206 21621 raft_consensus.cc:2388] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:48.966512 22872 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.054665 22772 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.031759 22704 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:48.922137 22885 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20250114 08:58:49.064224 21548 raft_consensus.cc:2363] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:49.038189 22786 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:49.059702 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 87574.8us
I20250114 08:58:49.047103 21623 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:49.033540 22821 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250114 08:58:48.969200 22875 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.972312 22823 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.038125 21555 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:49.072856 21554 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:49.039465 21553 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:49.074162 21552 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:49.075502 21594 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:48.945011 22712 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:49.047468 22873 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:49.049953 22838 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.059221 22886 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.059166 22715 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.060770 22818 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:49.051740 22820 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.057440 22850 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.065260 22832 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.065060 22835 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:48.968956 22864 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.051882 22742 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.056627 22785 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:48.972162 22874 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.057919 22808 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.042309 22876 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:49.081879 21551 rpcz_store.cc:269] 0114 08:58:46.924330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.927026 (+ 2696us) service_pool.cc:225] Handling call
0114 08:58:49.037370 (+2110344us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b5400194e54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:49.037497 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":976106368}
I20250114 08:58:49.269402 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.273465 22272 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.275310 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.246840 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:49.276855 21629 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:49.059124 22270 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
W20250114 08:58:49.270437 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 195012us
I20250114 08:58:49.289449 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.290278 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.290458 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.291126 21553 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.280193 21625 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:49.292021 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.292661 21553 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:49.293401 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.294346 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.294942 21555 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:49.295732 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.285794 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.297089 21552 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:49.297303 21594 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.297865 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.298333 21552 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:49.298878 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.299620 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.300066 21555 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:49.300688 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:49.301784 21593 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.305624 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.318718 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:49.308349 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.073650 22723 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:49.319528 21548 raft_consensus.cc:2388] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:58:49.142292 21651 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5480 after lost signal to thread 20385
I20250114 08:58:49.247226 22690 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.318858 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.335137 21554 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:49.320372 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.320786 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.321930 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.323021 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.323323 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.321214 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.337188 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.230109 22847 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.210886 22794 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.161381 22870 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.324129 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.328984 21549 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:49.079402 22834 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.316066 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.231184 22857 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.064523 22240 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.197685 22883 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.273558 22854 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.333586 21593 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:49.523744 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 190229us
I20250114 08:58:49.284641 22831 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.331106 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.524886 21552 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.525753 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.420269 22866 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.436381 22727 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.230911 22843 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.142316 22891 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.303052 22873 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.069262 22881 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.142105 22882 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.276726 22886 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.067407 22848 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:49.342635 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.545379 21622 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.546307 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.343604 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.332741 22840 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.197s
I20250114 08:58:49.344306 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.344563 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.344748 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.550129 21627 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:49.346503 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.346742 22862 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.197367 22766 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.155850 22829 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.287667 22892 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.347147 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.550997 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.551345 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.323827 22772 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:49.480743 21651 debug-util.cc:398] Leaking SignalData structure 0x7b0800196be0 after lost signal to thread 21487
I20250114 08:58:49.197765 22673 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:49.480571 22704 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.364954 22723 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:49.475458 22853 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:49.353513 21669 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.019s sys 0.024s
W20250114 08:58:49.562789 21669 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.725s user 0.023s sys 0.041s
W20250114 08:58:49.563143 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 725661us
I20250114 08:58:49.451156 22818 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.483325 21550 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:58:49.569732 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.483541 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.413194 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.573089 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.574182 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.574771 21550 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:49.575503 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:49.485677 21546 consensus_meta.cc:287] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.627s user 0.008s sys 0.002s
I20250114 08:58:49.576108 21546 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:49.576663 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 43) took 1349 ms. Trace:
I20250114 08:58:49.485329 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.486331 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 45) took 1257 ms. Trace:
I20250114 08:58:49.487607 22823 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.584836 22848 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.348824 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.549142 22891 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:49.582692 22673 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:49.586172 21623 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.586503 22673 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:49.587224 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.587383 22891 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:49.413312 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 90014.5us
I20250114 08:58:49.587882 21623 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.351361 21629 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.588722 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.589483 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.590118 21629 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.590958 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.591516 21629 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:49.576989 21546 rpcz_store.cc:276] 0114 08:58:48.227337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:48.227471 (+ 134us) service_pool.cc:225] Handling call
0114 08:58:49.576598 (+1349127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:49.395253 22861 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.323341 22835 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.494931 22856 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.498833 22874 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.476204 22814 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.505118 22742 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.498905 22712 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.506832 22667 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.509114 22863 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.488915 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.602320 22835 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.607892 22835 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.608314 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.609592 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.486125 22871 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.418802 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.612347 21704 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.603288 22673 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.394287 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.513942 21555 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:58:49.514303 21549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1001) took 2585 ms (client timeout 1953 ms). Trace:
W20250114 08:58:49.615363 21549 rpcz_store.cc:269] 0114 08:58:46.928561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:46.928725 (+ 164us) service_pool.cc:225] Handling call
0114 08:58:49.514213 (+2585488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:49.515324 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.516291 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.301484 22796 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.526360 21552 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:49.616942 22876 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.617275 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.617408 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:49.612859 21651 debug-util.cc:398] Leaking SignalData structure 0x7b0800398340 after lost signal to thread 21579
I20250114 08:58:49.617950 21555 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.618055 21552 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:58:49.618815 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.618996 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.619221 22875 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.619413 21555 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.337800 21703 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.420495 22858 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.337800 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:49.527884 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 471051us
I20250114 08:58:49.348268 22832 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.452611 22837 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.306582 22693 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:49.530678 22873 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:49.547281 21628 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.343721 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.546851 21622 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.526839 22854 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.528399 22857 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.206148 22868 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.346199 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.372378 22239 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.551689 21627 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.551998 21701 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.535132 22730 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.620138 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:49.619011 21651 debug-util.cc:398] Leaking SignalData structure 0x7b080028c0c0 after lost signal to thread 21654
I20250114 08:58:49.620185 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:49.557731 21520 leader_election.cc:400] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:49.559841 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.664996 21520 leader_election.cc:403] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:49.665257 21700 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.665380 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:49.665839 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 123229us
I20250114 08:58:49.666098 22854 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:49.529642 22883 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.666538 22839 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:49.666625 21520 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:49.667011 22883 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:49.667253 21651 debug-util.cc:398] Leaking SignalData structure 0x7b0800178720 after lost signal to thread 21728
I20250114 08:58:49.452059 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.565248 22704 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:49.668099 21698 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.575037 22723 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.485937 22734 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.547885 22843 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.577958 21550 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.680610 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.582680 21547 rpcz_store.cc:276] 0114 08:58:48.228864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:48.228981 (+ 117us) service_pool.cc:225] Handling call
0114 08:58:49.486155 (+1257174us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b540016ec54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:49.486272 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":691672832}
I20250114 08:58:49.583967 22881 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.584595 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.585578 21696 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.490572 21548 raft_consensus.cc:2388] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:49.291071 22786 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:49.592303 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.598318 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.581799 22766 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.704646 22269 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:49.705245 22766 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.704677 22786 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:49.708009 22766 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.697081 22899 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.708863 22786 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.599237 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.555603 22892 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.605456 22878 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.556399 22882 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.616276 21554 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.664232 22857 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.669414 21552 raft_consensus.cc:2405] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20250114 08:58:49.684635 22839 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:49.670025 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 324124us
I20250114 08:58:49.614897 21626 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
W20250114 08:58:49.556320 21668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.000s sys 0.023s
W20250114 08:58:49.770583 21668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.000s sys 0.023s
I20250114 08:58:49.628988 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:49.770932 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 717592us
I20250114 08:58:49.771672 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.772106 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.772186 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.772846 22272 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:58:49.773248 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.773841 21626 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:49.671344 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.669847 21551 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:49.672389 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.774701 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.775102 21700 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:49.775823 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.775924 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.774716 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.647279 21704 pending_rounds.cc:77] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 5
I20250114 08:58:49.668409 21625 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.580708 22855 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.667891 22730 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.555809 22772 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.681537 21628 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.786134 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.681749 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.682169 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:58:49.682432 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 66477.7us
I20250114 08:58:49.683378 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.699944 21629 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:49.699254 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.787375 21550 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
W20250114 08:58:49.787767 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 105683us
I20250114 08:58:49.788322 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.699847 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.789260 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.702215 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.631425 22864 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.628038 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.671012 22832 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.832973 21521 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.799814 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.781213 21546 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:49.684774 22712 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.680656 22796 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:49.624864 21553 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:58:49.769311 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.578467 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.641958 22833 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.778498 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.785676 22730 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.782744 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.853221 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.853528 21594 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.853657 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:49.768486 21651 debug-util.cc:398] Leaking SignalData structure 0x7b08002c5ae0 after lost signal to thread 22299
I20250114 08:58:49.710767 22872 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.784832 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.789846 21628 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:49.857903 21626 raft_consensus.cc:2405] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:49.858110 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.708969 22899 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.790366 21698 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:49.697058 22843 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.698028 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.769551 22857 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.834540 21521 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.866688 21521 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.866459 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.835139 22845 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.836483 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.803568 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.792881 22734 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250114 08:58:49.840279 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.792851 22840 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.788681 22772 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.807842 21668 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:49.895473 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 87722.1us
I20250114 08:58:49.843079 21667 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.845693 21625 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:49.777961 21704 pending_rounds.cc:99] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250114 08:58:49.846400 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.896101 21668 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:49.896564 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 53566.5us
I20250114 08:58:49.839329 22712 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.850921 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.852121 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.898954 21621 raft_consensus.cc:2405] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:49.852362 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.771080 22878 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:49.853966 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.899598 21550 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.854841 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.837514 22832 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:49.856120 21594 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.899839 21700 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
W20250114 08:58:49.902153 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 46106.8us
I20250114 08:58:49.856808 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.902145 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.790167 21520 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:49.698253 22900 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250114 08:58:49.904273 21553 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:49.905004 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.906144 21668 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.907397 21668 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.781903 22823 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20250114 08:58:49.788547 22766 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20250114 08:58:49.910990 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.891772 21521 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.891891 21701 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:49.857208 22882 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:49.890295 21626 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
W20250114 08:58:49.855398 21651 debug-util.cc:398] Leaking SignalData structure 0x7b08002fa4a0 after lost signal to thread 22300
I20250114 08:58:49.853796 21554 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:49.897364 21704 raft_consensus.cc:2883] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:58:49.855170 22892 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:49.873908 22693 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:49.898432 22715 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.903066 21625 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
W20250114 08:58:49.903672 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 113569us
I20250114 08:58:49.868638 22833 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.890715 22845 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.896759 22840 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:49.892517 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.914068 22861 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.896409 22820 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.914249 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.893889 22881 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.901412 22712 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.901859 22891 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.918418 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.916251 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:49.913959 22894 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.845376 22796 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:49.896970 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 219441us
I20250114 08:58:49.904634 22885 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:49.908492 22872 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.927038 21625 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:49.909397 22855 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:49.983924 22855 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.984385 22855 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:49.984591 22240 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.927045 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:49.948624 22820 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.931402 21628 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:49.993906 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.941532 21629 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:49.925041 22808 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:49.918870 22715 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.950456 21626 raft_consensus.cc:2405] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20250114 08:58:49.916548 21521 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:50.010165 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 93724.7us
I20250114 08:58:50.010848 21521 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:49.911146 22785 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250114 08:58:49.910810 22786 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 08:58:49.936233 22840 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:49.930754 22845 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:49.928151 22833 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:49.937815 22861 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.973143 21547 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
W20250114 08:58:49.989027 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 62065.7us
I20250114 08:58:49.989284 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:49.989805 21518 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:49.943650 22881 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.913187 21594 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:49.988675 22821 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:49.998030 21625 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:49.999078 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.999572 22690 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.002876 22857 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:50.004853 22903 log.cc:926] Time spent T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
W20250114 08:58:50.005602 22902 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20250114 08:58:50.005211 22855 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.005496 22892 raft_consensus.cc:3049] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:49.949661 22891 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.955057 22894 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:49.949644 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:50.013691 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.973699 22899 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.016690 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
W20250114 08:58:50.135275 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 118661us
I20250114 08:58:50.135979 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:49.989419 22673 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:58:50.139878 22727 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.118943 22892 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:50.140321 22727 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.140728 22727 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.141034 22892 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.142354 22892 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.019814 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:49.976894 22872 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.145015 22872 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.026733 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:49.943953 22886 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.011627 22808 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.006883 22820 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.034034 22833 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:50.058796 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 149178us
I20250114 08:58:50.059019 22882 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.058820 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:50.183578 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 124822us
I20250114 08:58:50.184863 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.131507 22894 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.186556 21703 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:50.186511 21520 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.012816 22715 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.005652 22878 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.065924 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:50.066601 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:50.067381 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:50.017053 22839 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.068109 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:50.070266 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 80568.1us
I20250114 08:58:50.016906 22835 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.082368 22840 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.088797 22843 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.089244 22821 raft_consensus.cc:695] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:50.090706 22796 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.092756 22873 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.055153 22861 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.101688 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:49.968251 22831 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.035591 22693 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:50.122776 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.124337 21627 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:50.125172 21555 raft_consensus.cc:604] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.063398 22896 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.065842 22881 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.022406 22850 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.069372 22690 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:50.146628 22272 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:50.225611 21549 raft_consensus.cc:2371] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3. Re-sending same reply.
I20250114 08:58:50.125540 22891 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.182225 21594 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:50.189172 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.260841 21518 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:50.198282 21626 raft_consensus.cc:1235] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:50.178028 22898 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250114 08:58:50.170719 22808 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.194172 21552 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:58:50.161875 22814 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250114 08:58:50.224373 22854 consensus_meta.cc:287] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.558s user 0.007s sys 0.003s
I20250114 08:58:50.185724 22882 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.113240 22824 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:50.191707 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:58:50.263166 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 81032.9us
I20250114 08:58:50.274176 21594 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:50.264564 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 75592.8us
I20250114 08:58:50.200621 22847 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.275328 21520 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.277021 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.200959 22839 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.278080 21520 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.265982 21699 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:50.280069 21553 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:50.200316 22888 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.267931 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.195418 22848 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:50.257035 22850 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.281817 21626 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:50.260155 22690 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.269704 22890 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.270251 22818 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:50.266866 22814 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:50.274549 22882 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.300927 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.308665 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.310140 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.278237 22847 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.284606 22839 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.291857 22723 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.288580 22885 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.286101 22888 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:50.310851 21700 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.289119 22824 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:50.289851 22850 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.262421 22693 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.336126 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.275091 22854 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.304056 22890 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:50.260407 22896 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.308377 22893 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:50.331305 22864 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.346796 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.315961 22847 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:50.351496 22241 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.021s (SENT) [suppressed 19 similar messages]
I20250114 08:58:50.348703 22855 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.351804 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:50.353765 22704 consensus_meta.cc:287] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.674s user 0.010s sys 0.000s
W20250114 08:58:50.354185 22907 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20250114 08:58:50.333865 22885 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:50.348434 22890 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.357051 22840 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:50.335728 22821 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.358112 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.361812 22693 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:50.369877 22854 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.336728 22723 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:50.433550 22723 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.378993 22832 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.439392 22894 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.441286 22832 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.444099 21622 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:50.368296 22872 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.445607 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.381439 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1200) took 1058 ms. Trace:
I20250114 08:58:50.384528 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:50.445607 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.450776 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.451565 21704 raft_consensus.cc:2363] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:50.455114 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.456346 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.457515 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.458163 21696 raft_consensus.cc:2363] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:50.458879 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.459875 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:50.460680 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 323830us
I20250114 08:58:50.447299 21699 rpcz_store.cc:276] 0114 08:58:49.322985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.323105 (+ 120us) service_pool.cc:225] Handling call
0114 08:58:50.381214 (+1058109us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54002ea254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:50.381369 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2135795968}
I20250114 08:58:50.452338 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.462996 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.463584 21699 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.464424 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.387481 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:52970 (request call id 0) took 1094 ms. Trace:
I20250114 08:58:50.328101 22857 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:50.381501 22831 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:50.307441 22814 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:50.394912 21630 raft_consensus.cc:2388] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:50.339419 22888 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.408372 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.393764 22864 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:58:50.432540 22906 log.cc:926] Time spent T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.348s user 0.005s sys 0.000s
I20250114 08:58:50.433439 22818 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.464851 21703 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:50.466254 21451 catalog_manager.cc:5526] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 2, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:58:50.504151 21451 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:50.383210 22839 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.382277 22850 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.467404 21626 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:50.461865 21452 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 70) took 3541 ms. Trace:
I20250114 08:58:50.469362 22239 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.506500 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.468374 21555 rpcz_store.cc:276] 0114 08:58:49.293499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.620045 (+326546us) service_pool.cc:225] Handling call
0114 08:58:50.387224 (+767179us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b5400323a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5c7bc3 00007fa64452e80a 00007fa63dc07d14 00007fa63dc07ca2 00007fa63dc07bd1 00007fa63dc07b4c 00007fa63dc06e02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:50.387361 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1048132480}
I20250114 08:58:50.471752 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.474395 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.481423 22882 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.483310 22894 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.408905 22885 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.502097 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.491693 21669 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:50.535194 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 43608.3us
I20250114 08:58:50.535589 21451 catalog_manager.cc:5526] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 2, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:50.536873 21669 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:50.537681 21451 catalog_manager.cc:5526] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 4, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:50.422772 22835 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.427335 22886 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:50.460399 22772 consensus_meta.cc:287] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.556s user 0.006s sys 0.000s
I20250114 08:58:50.514173 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.543428 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.578804 21696 raft_consensus.cc:2371] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3. Re-sending same reply.
I20250114 08:58:50.579957 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.563846 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.564500 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.465881 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.568569 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1117) took 1225 ms. Trace:
I20250114 08:58:50.580804 21696 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.601482 21630 rpcz_store.cc:276] 0114 08:58:49.342899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.344535 (+ 1636us) service_pool.cc:225] Handling call
0114 08:58:50.568256 (+1223721us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b54002d05d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:50.568483 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1930596096}
I20250114 08:58:50.569514 21621 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:50.602388 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.452163 22861 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.600792 21703 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.570788 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.603776 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.490816 22673 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.616102 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.498795 22864 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.506278 21452 rpcz_store.cc:276] 0114 08:58:46.920783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:47.479678 (+558895us) service_pool.cc:225] Handling call
0114 08:58:50.461787 (+2982109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:48.871145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:48.946910 (+ 75765us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:48.946945 (+ 35us) write_op.cc:435] Acquired schema lock
0114 08:58:48.946956 (+ 11us) tablet.cc:659] Decoding operations
0114 08:58:48.961752 (+ 14796us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:48.961819 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:48.967261 (+ 5442us) tablet.cc:682] Acquiring locks for 6 operations
0114 08:58:49.061921 (+ 94660us) tablet.cc:698] Row locks acquired
0114 08:58:49.065194 (+ 3273us) write_op.cc:260] PREPARE: finished
0114 08:58:49.084875 (+ 19681us) write_op.cc:270] Start()
0114 08:58:49.145381 (+ 60506us) write_op.cc:276] Timestamp: P: 1736845129080594 usec, L: 0
0114 08:58:49.150962 (+ 5581us) op_driver.cc:348] REPLICATION: starting
0114 08:58:49.284290 (+133328us) log.cc:844] Serialized 2533 byte log entry
0114 08:58:49.780372 (+496082us) op_driver.cc:464] REPLICATION: finished
0114 08:58:49.924823 (+144451us) write_op.cc:301] APPLY: starting
0114 08:58:49.924911 (+ 88us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:49.941051 (+ 16140us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:49.943349 (+ 2298us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:50.063490 (+120141us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:50.078212 (+ 14722us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0114 08:58:50.078261 (+ 49us) write_op.cc:312] APPLY: finished
0114 08:58:50.078532 (+ 271us) log.cc:844] Serialized 52 byte log entry
0114 08:58:50.078709 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:50.078826 (+ 117us) write_op.cc:454] Released schema lock
0114 08:58:50.078874 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:50.079218 (+ 344us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:50.079241 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":48959,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":140628,"apply.run_cpu_time_us":4188,"apply.run_wall_time_us":154365,"num_ops":6,"prepare.queue_time_us":49182,"prepare.run_cpu_time_us":13019,"prepare.run_wall_time_us":678277,"raft.queue_time_us":145206,"raft.run_cpu_time_us":12289,"raft.run_wall_time_us":213950,"replication_time_us":563419,"thread_start_us":301495,"threads_started":4,"wal-append.queue_time_us":226354}]]}
I20250114 08:58:50.612323 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57308 (request call id 447) took 1037 ms. Trace:
I20250114 08:58:50.604426 21696 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.603276 21700 raft_consensus.cc:2371] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3. Re-sending same reply.
I20250114 08:58:50.623453 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.534405 22818 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:50.456267 22715 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:50.626325 21550 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.004s sys 0.007s
W20250114 08:58:50.635152 21550 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.514s user 0.004s sys 0.007s
I20250114 08:58:50.631800 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1002) took 1061 ms. Trace:
I20250114 08:58:50.623904 21518 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:50.624572 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 420964us
I20250114 08:58:50.638175 21704 raft_consensus.cc:2363] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:50.636314 21621 rpcz_store.cc:276] 0114 08:58:49.570744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.029609 (+458865us) service_pool.cc:225] Handling call
0114 08:58:50.569253 (+539644us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:50.605579 (+ 36326us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:50.631483 (+ 25904us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54002d05d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:50.631694 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1240050560}
I20250114 08:58:50.595886 22899 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 08:58:50.641119 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 309626us
W20250114 08:58:50.641677 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 256391us
I20250114 08:58:50.643951 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.595753 22823 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.650771 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.651826 21699 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.623544 21627 rpcz_store.cc:276] 0114 08:58:49.545638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.542179 (+996541us) service_pool.cc:225] Handling call
0114 08:58:50.583138 (+ 40959us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:50.654798 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.605360 22768 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:58:50.655586 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet 3b369b35736c413ca069db061ed36bc5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d7732324fb5f48d3b16a29d4bb2584ed is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:58:50.660954 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 382922us
I20250114 08:58:50.659977 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.661644 22272 raft_consensus.cc:1235] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:50.620075 22673 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:50.665665 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:50.691386 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 67570.3us
I20250114 08:58:50.685402 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.619753 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:50.697006 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:50.681484 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 212171us
I20250114 08:58:50.708804 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:50.704160 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.711349 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:50.706066 21520 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.711948 21555 raft_consensus.cc:604] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.712085 22239 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:50.713510 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.706981 21700 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 1.
I20250114 08:58:50.607630 22772 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.654486 22823 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.714641 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:50.715155 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 441015us
I20250114 08:58:50.725517 22878 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:50.726154 22878 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.728253 22878 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.648717 22899 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.729904 22899 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.731590 21699 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:50.665978 22873 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:50.722265 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.711524 21696 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.725493 21594 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:50.697753 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:50.739355 21520 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.740495 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:50.626278 22848 raft_consensus.cc:695] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:50.694861 22840 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:50.694622 22704 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:50.704960 22693 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:50.715061 22730 consensus_meta.cc:287] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.844s user 0.007s sys 0.006s
I20250114 08:58:50.707338 22768 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:50.654590 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 306544us
I20250114 08:58:50.770941 21628 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:58:50.780520 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 82863.2us
I20250114 08:58:50.782918 22848 consensus_queue.cc:237] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.733929 22823 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:50.758445 22883 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.091s user 0.009s sys 0.005s
W20250114 08:58:50.856528 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 131099us
I20250114 08:58:50.732579 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:50.861068 21594 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:50.862270 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.864671 22872 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.864784 22835 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:50.865010 22881 consensus_meta.cc:287] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.600s user 0.008s sys 0.001s
I20250114 08:58:50.865326 22835 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.865782 22835 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.865492 22881 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.867800 22881 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.855667 22730 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.865732 22715 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.757896 22914 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:50.876945 22883 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.882256 22883 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.876981 22840 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.878532 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:50.879192 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:50.883512 21621 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:50.883659 22840 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:50.884161 22840 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:50.864812 22933 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:50.884871 22933 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.885406 22933 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:50.885152 22847 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.892206 21624 raft_consensus.cc:2388] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:58:50.897056 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 185102us
I20250114 08:58:50.870462 22730 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:50.865864 22916 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
W20250114 08:58:50.905752 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 44809.8us
I20250114 08:58:50.904887 22239 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:50.905788 21596 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.913255 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:50.913123 21594 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:50.919970 21596 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.920764 21594 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.858789 22921 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.859925 22896 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:50.900224 22881 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:58:50.918777 22845 consensus_meta.cc:287] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.862s user 0.005s sys 0.005s
I20250114 08:58:50.897142 22837 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.904863 22883 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:50.951493 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
W20250114 08:58:50.950994 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 188120us
I20250114 08:58:50.898633 21695 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:50.788651 22704 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:50.930572 22857 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.944185 22266 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:58:50.930035 21596 leader_election.cc:400] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:50.922755 22831 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:50.931011 22832 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.908856 22693 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:50.863096 22768 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:50.945719 22712 consensus_meta.cc:287] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.991s user 0.011s sys 0.000s
I20250114 08:58:50.952080 22839 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.014617 21596 leader_election.cc:403] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:58:50.944890 22921 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:50.951332 22850 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.922516 22854 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 08:58:51.039952 21624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1107) took 2007 ms (client timeout 1893 ms). Trace:
I20250114 08:58:51.039777 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:51.045529 21629 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
W20250114 08:58:50.921787 22727 consensus_meta.cc:287] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.781s user 0.018s sys 0.001s
W20250114 08:58:51.048856 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 127150us
I20250114 08:58:51.015086 22832 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:51.055857 22820 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.870s user 0.007s sys 0.003s
W20250114 08:58:51.016701 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 46609.6us
I20250114 08:58:50.968287 22872 consensus_queue.cc:1046] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
I20250114 08:58:50.940095 22926 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.014967 22882 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:50.983968 22919 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:50.983055 22857 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:51.056475 22921 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:50.931661 22730 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.018218 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:51.042732 22896 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:51.064213 21596 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:50.975909 22845 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:50.973920 22861 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:51.069383 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 120635us
I20250114 08:58:51.075524 22823 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.046651 22693 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:51.054771 22831 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:51.017690 22883 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
W20250114 08:58:51.060999 21624 rpcz_store.cc:269] 0114 08:58:49.032110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.032711 (+ 601us) service_pool.cc:225] Handling call
0114 08:58:51.039656 (+2006945us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54002df5d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:51.039861 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4191668608}
W20250114 08:58:51.078653 22912 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.418s user 0.000s sys 0.001s
W20250114 08:58:51.090399 22928 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20250114 08:58:51.094125 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 62) took 1103 ms. Trace:
I20250114 08:58:51.068640 22832 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.098678 21596 leader_election.cc:400] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
W20250114 08:58:51.100138 22917 log.cc:926] Time spent T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.209s user 0.000s sys 0.000s
I20250114 08:58:51.067543 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:51.078405 22772 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:51.089182 22723 consensus_meta.cc:287] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.652s user 0.007s sys 0.002s
I20250114 08:58:51.062397 22939 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:51.086901 22847 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.087757 22857 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.113132 21594 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.118624 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1008) took 1538 ms. Trace:
I20250114 08:58:51.050653 22712 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.085968 22919 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:51.124419 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:51.069319 22727 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.102362 22882 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.076768 22820 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.113386 22241 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.151582 22899 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
W20250114 08:58:51.111093 22855 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.001s sys 0.039s
I20250114 08:58:51.155150 22864 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.157279 21596 leader_election.cc:403] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:58:51.206379 22772 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:51.237696 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 139132us
I20250114 08:58:51.239286 21596 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:51.162143 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.163980 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:51.169940 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 56874.1us
W20250114 08:58:51.157248 22888 consensus_meta.cc:287] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.655s user 0.009s sys 0.000s
W20250114 08:58:51.158409 22890 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.709s user 0.002s sys 0.008s
I20250114 08:58:51.125941 22693 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:51.217027 22931 log.cc:926] Time spent T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
W20250114 08:58:51.218261 22942 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250114 08:58:51.220125 22920 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:51.220561 22847 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:58:51.221009 22938 log.cc:926] Time spent T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250114 08:58:51.166397 22723 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.221427 22768 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:51.223721 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 105422us
I20250114 08:58:51.170727 22939 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:51.209162 22690 consensus_meta.cc:287] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.905s user 0.017s sys 0.004s
I20250114 08:58:51.169486 21629 rpcz_store.cc:276] 0114 08:58:49.579886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.145259 (+565373us) service_pool.cc:225] Handling call
0114 08:58:51.040433 (+895174us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:51.063192 (+ 22759us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:51.118363 (+ 55171us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54002df5d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:51.118544 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2057785600}
I20250114 08:58:51.234046 22272 raft_consensus.cc:2363] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:51.154498 21622 rpcz_store.cc:276] 0114 08:58:49.991047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.466335 (+475288us) service_pool.cc:225] Handling call
0114 08:58:50.466648 (+ 313us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:50.466996 (+ 348us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:50.467023 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:50.606035 (+139012us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b180015d360. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c661ec6 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63dfae741 00007fa63dfbdd72 00007fa63f609141 00007fa63f5cd58a
0114 08:58:50.607976 (+ 1941us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b2c00046c78. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63dfae741 00007fa63dfbdd72 00007fa63f609141 00007fa63f5cd58a 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782
0114 08:58:50.623071 (+ 15095us) log.cc:844] Serialized 26 byte log entry
0114 08:58:50.856929 (+233858us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:50.856968 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:50.857719 (+ 751us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:51.093890 (+236171us) raft_consensus.cc:1680] finished
0114 08:58:51.093911 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:51.094063 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":198,"raft.queue_time_us":3809,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":6073,"spinlock_wait_cycles":53295232,"thread_start_us":197303,"threads_started":1,"wal-append.queue_time_us":350911}
I20250114 08:58:51.253407 22937 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:58:51.241636 21596 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.242275 21594 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:51.205511 22919 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.222395 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.283282 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.254408 21700 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:51.112058 22896 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.253144 22831 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.236357 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 41) took 1563 ms. Trace:
I20250114 08:58:51.248438 22772 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:51.247188 22925 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:51.265836 22271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 186) took 1346 ms. Trace:
I20250114 08:58:51.236796 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.257056 22882 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:51.278977 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:51.268839 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 171815us
I20250114 08:58:51.282651 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.282999 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.267261 22269 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:58:51.232503 22855 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.834s user 0.001s sys 0.040s
I20250114 08:58:51.266680 22920 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.285084 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.285666 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.306870 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.306979 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.280637 21596 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.291894 21697 rpcz_store.cc:276] 0114 08:58:49.673046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.844282 (+171236us) service_pool.cc:225] Handling call
0114 08:58:49.844603 (+ 321us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:49.844914 (+ 311us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:49.844941 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:49.855921 (+ 10980us) log.cc:844] Serialized 26 byte log entry
0114 08:58:50.036084 (+180163us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b18000e2ba0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:58:50.257930 (+221846us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:50.257966 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:50.258030 (+ 64us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:51.229347 (+971317us) raft_consensus.cc:1680] finished
0114 08:58:51.229366 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:51.236292 (+ 6926us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3964,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":9087,"spinlock_wait_cycles":18670848,"thread_start_us":221641,"threads_started":1,"wal-append.queue_time_us":675883}
I20250114 08:58:51.284399 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 109) took 1279 ms. Trace:
I20250114 08:58:51.299155 22882 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:51.326606 22882 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.326239 22857 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.326925 22899 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:51.327070 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.293249 22835 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.301683 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1226) took 1277 ms. Trace:
I20250114 08:58:51.256733 22888 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.328260 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.293265 21702 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:51.328703 21704 rpcz_store.cc:276] 0114 08:58:50.024293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.710108 (+685815us) service_pool.cc:225] Handling call
0114 08:58:50.741270 (+ 31162us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:50.741601 (+ 331us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:50.741625 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:50.775219 (+ 33594us) log.cc:844] Serialized 26 byte log entry
0114 08:58:50.987118 (+211899us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:50.987174 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:50.987269 (+ 95us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:51.301496 (+314227us) raft_consensus.cc:1680] finished
0114 08:58:51.301518 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:51.301632 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2369,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":24990,"thread_start_us":79393,"threads_started":1,"wal-append.queue_time_us":286464}
I20250114 08:58:51.330781 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.332499 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.254370 22890 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.337081 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.271847 22847 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.300954 21703 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:51.293668 22673 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.583s user 0.009s sys 0.001s
I20250114 08:58:51.294632 22772 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.308941 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.310343 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.284082 22943 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:51.296842 22271 rpcz_store.cc:276] 0114 08:58:49.924420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.926387 (+ 1967us) service_pool.cc:225] Handling call
0114 08:58:51.236441 (+1310054us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b5400710554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:51.254697 (+ 18256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1661131136}
I20250114 08:58:51.313587 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:51.314810 22864 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.318831 22843 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.320123 22878 raft_consensus.cc:3049] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:51.320827 22823 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:51.322120 22935 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.226s user 0.000s sys 0.003s
I20250114 08:58:51.322259 22896 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:51.324647 21695 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:51.368346 22890 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.369403 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1228) took 1344 ms. Trace:
I20250114 08:58:51.369834 21700 rpcz_store.cc:276] 0114 08:58:50.024965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.715195 (+690230us) service_pool.cc:225] Handling call
0114 08:58:50.715471 (+ 276us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:51.287177 (+571706us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:51.369158 (+ 81981us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400327654. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:51.369342 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239041792}
W20250114 08:58:51.371443 22885 consensus_meta.cc:287] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.774s user 0.004s sys 0.004s
I20250114 08:58:51.280839 22690 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:51.324965 22814 consensus_meta.cc:287] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.819s user 0.010s sys 0.000s
I20250114 08:58:51.360774 22832 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.372977 22814 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.374150 22690 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.374809 22275 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:51.375026 22814 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:51.298964 22925 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.376754 21552 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.308159 22854 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:51.378077 22730 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.378295 22854 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:51.378731 22854 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:51.378718 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.332582 22888 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.303179 21669 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:51.378083 22925 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:51.381304 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 78199.6us
I20250114 08:58:51.383806 22896 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:51.384274 22896 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:58:51.384744 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 45157.9us
I20250114 08:58:51.286244 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.333190 21624 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.333679 21550 raft_consensus.cc:2363] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:51.284759 22704 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:51.334668 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.334822 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.399834 21553 raft_consensus.cc:2405] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:51.335912 21518 leader_election.cc:400] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Vote denied by peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045) with higher term. Message: Invalid argument: T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:51.281845 22883 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.336529 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.398939 21628 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.407649 21627 raft_consensus.cc:2388] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:51.308800 21697 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:51.284902 22848 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:51.341866 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.345300 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.346840 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.349370 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:51.414718 21554 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.353266 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:51.416193 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 74380.1us
I20250114 08:58:51.416770 21626 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.417496 21596 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.417644 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.418258 21626 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.418706 22690 consensus_queue.cc:1046] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:51.419080 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.419870 22690 raft_consensus.cc:3049] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:51.420293 22690 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:51.420807 22723 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.421402 22864 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.422235 22835 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:51.422667 22952 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
W20250114 08:58:51.326269 22833 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.137s user 0.011s sys 0.000s
I20250114 08:58:51.424942 22833 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.425215 22939 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.425292 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.425982 21553 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.426932 22833 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.332005 22840 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.352105 22941 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:51.429365 22840 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.430294 21625 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:51.421185 22690 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:51.431591 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1006) took 1852 ms. Trace:
I20250114 08:58:51.306388 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.360780 21630 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:51.432070 22847 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.431936 21625 rpcz_store.cc:276] 0114 08:58:49.579581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.066957 (+487376us) service_pool.cc:225] Handling call
0114 08:58:50.067214 (+ 257us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:51.430807 (+1363593us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:51.431365 (+ 558us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002de1d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:51.431536 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3134208384}
I20250114 08:58:51.433331 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.434036 21625 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.434213 21696 raft_consensus.cc:2363] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:51.434381 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.435142 21630 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.361270 21548 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:51.362246 22712 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.305550 22845 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:51.364845 22818 consensus_meta.cc:287] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.718s user 0.008s sys 0.003s
I20250114 08:58:51.462925 22818 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.328321 22857 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.365835 22933 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.389039 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:51.465238 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 76310.8us
I20250114 08:58:51.465919 21669 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:51.371851 22885 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.466866 22818 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:51.467016 22933 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.468676 22823 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:51.469127 22823 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:51.469480 22823 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:51.470399 22274 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:51.400602 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.387319 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.404801 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.467046 21669 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:51.481637 21624 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.482707 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.483444 21624 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:51.345448 22943 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:51.384745 22896 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.408427 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.408430 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.490700 21628 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 5
I20250114 08:58:51.410979 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.491413 21550 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:51.419704 22925 consensus_queue.cc:1046] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:51.437671 21630 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:51.414003 21555 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:51.431890 22690 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.457562 21625 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.352885 22878 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:51.459543 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.470494 22820 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.400579 21518 leader_election.cc:403] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:51.374852 22832 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.484387 21624 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:51.511453 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:51.485301 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 71607.6us
I20250114 08:58:51.490703 21627 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.492303 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:51.492962 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 75539.6us
I20250114 08:58:51.491071 22861 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:51.429387 22920 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.522058 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.523025 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.523881 21547 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:51.525019 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.526487 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.528394 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.490218 22943 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.448909 22727 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:51.435423 22831 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:51.504493 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.471289 21621 raft_consensus.cc:2363] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
W20250114 08:58:51.508734 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 172909us
I20250114 08:58:51.449033 22848 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.539345 21554 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.540624 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.541383 21554 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:51.542133 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.542685 21554 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.542941 21594 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.515496 21596 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:51.486559 22885 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.460234 22854 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.516309 21521 leader_election.cc:400] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:51.548275 21521 leader_election.cc:403] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20250114 08:58:51.501633 22837 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.517s user 0.010s sys 0.000s
I20250114 08:58:51.529253 21550 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.353525 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.551498 21629 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.493640 22899 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.460067 22883 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20250114 08:58:51.535749 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.535841 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:51.536537 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.562312 21624 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.562702 21627 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.512238 21555 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:51.513736 22891 consensus_meta.cc:287] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.243s user 0.008s sys 0.004s
I20250114 08:58:51.351078 22673 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.544845 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.570577 21548 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.503541 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.325896 22267 rpcz_store.cc:276] 0114 08:58:50.020626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.035273 (+ 14647us) service_pool.cc:225] Handling call
0114 08:58:50.072209 (+ 36936us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:50.293105 (+220896us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:50.296845 (+ 3740us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:50.333728 (+ 36883us) log.cc:844] Serialized 26 byte log entry
0114 08:58:50.768683 (+434955us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:58:50.778902 (+ 10219us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:50.781221 (+ 2319us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:51.258422 (+477201us) raft_consensus.cc:1680] finished
0114 08:58:51.258444 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:51.274654 (+ 16210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16772,"raft.queue_time_us":38179,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":18261,"thread_start_us":561989,"threads_started":2,"wal-append.queue_time_us":635245}
I20250114 08:58:51.522006 22861 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:51.559659 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:51.564858 21549 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.599547 22275 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.505201 22768 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20250114 08:58:51.573180 21628 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
W20250114 08:58:51.574412 21668 leader_election.cc:336] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.661s (SENT)
I20250114 08:58:51.578032 22926 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.578095 22953 raft_consensus.cc:3049] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:51.603236 22953 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:51.604001 21548 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.545185 22818 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045) with higher term. Message: Invalid argument: T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
I20250114 08:58:51.610868 21552 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:51.581815 21594 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:51.612061 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.613245 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:51.613451 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:51.614146 21700 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:51.613495 21624 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:51.591585 21627 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:51.615114 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:51.565615 21626 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.601502 22848 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:51.556646 22855 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:51.549722 22857 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:51.606441 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 101080us
I20250114 08:58:51.555330 22837 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.607482 21701 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:51.536864 22893 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250114 08:58:51.609356 21697 raft_consensus.cc:1270] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:51.615928 21700 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:51.613862 21703 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:58:51.597523 22864 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.617653 22883 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.608623 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.620785 21596 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.637763 21549 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:51.638211 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.638490 21596 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.638945 21552 raft_consensus.cc:2363] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5.
I20250114 08:58:51.639930 21596 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.621155 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.640362 22239 leader_election.cc:400] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5.
I20250114 08:58:51.596179 22861 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.645493 22275 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.641705 21594 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:51.622654 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.624819 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:51.646585 21703 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:51.615736 22845 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:51.648207 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.627952 21550 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.628736 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.649623 21700 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.649369 21593 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:51.640843 21696 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.640856 22239 leader_election.cc:403] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250114 08:58:51.644699 22960 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.614145 21554 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.652987 21594 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:51.588403 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 60) took 1597 ms. Trace:
I20250114 08:58:51.604116 22953 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.648432 21551 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:51.574818 22891 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.625084 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.640810 21596 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:51.662851 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:51.651041 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.653790 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.663784 21550 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.653944 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.618886 22855 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:51.622306 22857 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:51.657203 21594 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:51.659144 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.668069 21699 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:51.664862 21552 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:51.666350 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:51.666396 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:51.669107 21624 raft_consensus.cc:2363] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5.
I20250114 08:58:51.669283 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.670083 21552 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:51.669886 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.670755 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.595431 22896 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.673372 22953 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.665648 22855 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.675887 22941 raft_consensus.cc:2758] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250114 08:58:51.665374 22923 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.656286 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.660387 22832 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.578866 22878 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.680900 21698 raft_consensus.cc:2363] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:51.680781 22960 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.683643 21668 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:51.652490 21547 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:51.717773 21668 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:51.691514 21596 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.697955 22843 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.684672 21553 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.723168 21668 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:51.723974 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.646517 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:51.651594 22959 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:51.696949 22808 consensus_meta.cc:287] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.422s user 0.009s sys 0.004s
W20250114 08:58:51.706694 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 69672.3us
I20250114 08:58:51.656673 22723 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:51.678483 22891 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:51.658598 22883 raft_consensus.cc:695] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:51.654448 22958 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.711412 21521 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:51.683809 22857 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:51.695163 21555 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:51.755029 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.755756 21555 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:51.705152 22833 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:51.756815 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.757063 22833 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.757462 21555 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:51.758323 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.690948 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.724489 21596 leader_election.cc:400] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:51.759991 21596 leader_election.cc:403] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:58:51.714146 22923 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:51.726233 22673 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:51.720311 22240 leader_election.cc:400] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5.
I20250114 08:58:51.655798 22845 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.730190 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:51.746028 22848 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:51.743398 21696 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:51.697258 22882 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.672358 22772 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.750761 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:51.880751 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 130089us
I20250114 08:58:51.761643 21596 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:51.758893 21555 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:51.724833 21668 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:51.763350 21594 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:51.731633 22963 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:51.813212 22275 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:51.733793 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.744140 22960 raft_consensus.cc:695] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:51.712052 22919 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.892390 22899 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:51.869930 22843 raft_consensus.cc:695] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:51.897666 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.898752 21549 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:51.901412 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.902204 21548 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:51.903580 22823 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.904831 22940 raft_consensus.cc:988] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:58:51.899757 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:51.759608 22925 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:51.805550 22833 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:51.731384 22920 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3.
W20250114 08:58:51.882303 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 157560us
I20250114 08:58:51.806351 22820 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20250114 08:58:51.888274 22274 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:58:51.976734 21668 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:51.759779 22854 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:51.910049 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 179972us
I20250114 08:58:51.816849 22891 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:51.965708 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.967828 21548 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:58:51.967756 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.990584 21550 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:51.909554 22837 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:51.770273 22832 raft_consensus.cc:695] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:58:51.968501 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 81073.6us
I20250114 08:58:51.998970 21521 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:51.815397 22967 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:51.975140 21704 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:58:51.986757 22963 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:51.978197 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.908846 22923 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.978546 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.876045 22848 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:51.980041 21668 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:52.019809 21555 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:58:51.814080 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 116571us
W20250114 08:58:51.985725 21698 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 964) took 2054 ms (client timeout 1766 ms). Trace:
I20250114 08:58:51.746732 22808 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:51.969245 22940 raft_consensus.cc:1076] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Signalling peer cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:58:51.986361 22673 consensus_queue.cc:1046] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
W20250114 08:58:51.991964 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 228696us
I20250114 08:58:51.991240 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:51.966832 22831 raft_consensus.cc:3049] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
W20250114 08:58:51.977768 22898 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20250114 08:58:51.994868 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:52.010586 22969 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.974339 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
W20250114 08:58:52.007534 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 245942us
I20250114 08:58:52.040159 21596 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.017634 22963 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:51.977914 22712 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5.
I20250114 08:58:51.975051 21700 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:51.889922 22960 consensus_queue.cc:237] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:52.021404 21698 rpcz_store.cc:269] 0114 08:58:49.925200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.946710 (+ 21510us) service_pool.cc:225] Handling call
0114 08:58:51.978919 (+2032209us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54001ef0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:51.979040 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3726920064}
I20250114 08:58:52.050743 21700 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82151d61cef04050b69f24bc614507fc"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
I20250114 08:58:52.021935 21521 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:52.051357 21700 raft_consensus.cc:604] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.009202 22837 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:51.969875 22925 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:51.980763 22943 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.012116 22881 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:52.092658 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55864 (request call id 191) took 2197 ms. Trace:
I20250114 08:58:52.033699 22275 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:58:52.034044 21553 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:51.996861 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.097534 21549 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:52.010382 22968 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:52.008709 21550 raft_consensus.cc:1235] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.049171 22963 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.042086 22832 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.043634 22845 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:52.050413 22965 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:52.010077 22768 consensus_queue.cc:1046] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 08:58:51.974853 22883 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.046260 21596 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:58:52.115139 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 68968.6us
I20250114 08:58:52.115913 21596 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:52.091040 22837 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.016235 22723 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.054847 21521 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:52.123823 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 69047.6us
I20250114 08:58:52.092550 22969 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.055626 22893 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:58:52.032692 22896 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:52.044448 22690 consensus_meta.cc:287] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.534s user 0.007s sys 0.001s
I20250114 08:58:52.056648 21626 raft_consensus.cc:1235] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:52.059616 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:52.061303 22274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48992 (request call id 140) took 1131 ms. Trace:
W20250114 08:58:52.062167 22693 consensus_meta.cc:287] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.796s user 0.004s sys 0.004s
I20250114 08:58:52.123651 22266 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:52.023353 22843 consensus_queue.cc:237] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.071062 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:52.071853 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:52.079697 22882 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.083874 22878 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.086604 21548 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250114 08:58:52.017043 22891 consensus_queue.cc:1046] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250114 08:58:52.033522 22831 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.030328 21594 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:52.217113 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 186846us
I20250114 08:58:52.044652 22833 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:52.109648 21553 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:52.022545 22967 consensus_queue.cc:1046] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20250114 08:58:52.112079 21550 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:52.091905 21546 raft_consensus.cc:2363] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:52.110055 22919 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.118829 22835 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:51.658309 21623 rpcz_store.cc:276] 0114 08:58:49.990708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.453055 (+462347us) service_pool.cc:225] Handling call
0114 08:58:50.453410 (+ 355us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:50.453631 (+ 221us) raft_consensus.cc:1529] Early marking committed up to index 5
0114 08:58:50.708283 (+254652us) log.cc:844] Serialized 14 byte log entry
0114 08:58:51.405599 (+697316us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b180015d1e0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63f25808c 00007fa63c676bfc 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fe2b
0114 08:58:51.559262 (+153663us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:51.574064 (+ 14802us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:58:51.575995 (+ 1931us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:51.588094 (+ 12099us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:51.588312 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6844,"prepare.queue_time_us":464544,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":15383,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":73585920,"thread_start_us":588971,"threads_started":3,"wal-append.queue_time_us":394862}
I20250114 08:58:52.007071 22899 consensus_queue.cc:1046] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250114 08:58:52.091614 22925 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.119859 22854 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:58:52.103883 22881 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:52.055558 22818 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:52.198544 22893 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20250114 08:58:52.131435 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:52.244418 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 113060us
I20250114 08:58:52.245218 22239 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:52.047886 22269 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
W20250114 08:58:52.185442 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 125898us
I20250114 08:58:52.185745 21698 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:52.108186 22857 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.135684 22885 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:52.184221 22274 rpcz_store.cc:276] 0114 08:58:50.929590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:50.929730 (+ 140us) service_pool.cc:225] Handling call
0114 08:58:52.061150 (+1131420us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b540061e9d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:52.061250 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1138155264}
I20250114 08:58:52.231252 22970 raft_consensus.cc:3049] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:52.137717 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:52.197392 21699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:46398
I20250114 08:58:52.184429 22920 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:58:52.226511 22712 consensus_queue.cc:1046] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20250114 08:58:52.228382 22864 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20250114 08:58:52.222122 22833 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:52.206897 22768 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:52.230836 22271 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:52.228209 22973 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:52.117885 22965 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:52.298866 22965 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.195765 22693 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.302217 22693 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:58:52.302680 22953 consensus_meta.cc:287] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.598s user 0.004s sys 0.004s
I20250114 08:58:52.260986 21668 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:52.223151 22808 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.138044 22823 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.241490 22881 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:52.271534 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 276693us
I20250114 08:58:52.308348 21669 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.126101 22972 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:52.310581 21669 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:52.270608 21699 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:58:52.312686 21699 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.305171 22823 consensus_queue.cc:1046] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20250114 08:58:52.315021 22823 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:52.312031 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.098539 21451 rpcz_store.cc:276] 0114 08:58:49.894836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:49.895088 (+ 252us) service_pool.cc:225] Handling call
0114 08:58:52.092588 (+2197500us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:51.052978 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:51.090383 (+ 37405us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:51.090424 (+ 41us) write_op.cc:435] Acquired schema lock
0114 08:58:51.090444 (+ 20us) tablet.cc:659] Decoding operations
0114 08:58:51.091095 (+ 651us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:51.091161 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:51.091198 (+ 37us) tablet.cc:682] Acquiring locks for 4 operations
0114 08:58:51.091602 (+ 404us) tablet.cc:698] Row locks acquired
0114 08:58:51.091619 (+ 17us) write_op.cc:260] PREPARE: finished
0114 08:58:51.091771 (+ 152us) write_op.cc:270] Start()
0114 08:58:51.091878 (+ 107us) write_op.cc:276] Timestamp: P: 1736845131091724 usec, L: 0
0114 08:58:51.091901 (+ 23us) op_driver.cc:348] REPLICATION: starting
0114 08:58:51.113519 (+ 21618us) log.cc:844] Serialized 1712 byte log entry
0114 08:58:51.432833 (+319314us) op_driver.cc:464] REPLICATION: finished
0114 08:58:51.597050 (+164217us) write_op.cc:301] APPLY: starting
0114 08:58:51.604640 (+ 7590us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:51.647045 (+ 42405us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:51.661047 (+ 14002us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:51.728934 (+ 67887us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:51.729125 (+ 191us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0114 08:58:51.729170 (+ 45us) write_op.cc:312] APPLY: finished
0114 08:58:51.805244 (+ 76074us) log.cc:844] Serialized 40 byte log entry
0114 08:58:51.815690 (+ 10446us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:51.872128 (+ 56438us) write_op.cc:454] Released schema lock
0114 08:58:51.874511 (+ 2383us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:51.874959 (+ 448us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:51.875000 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26303,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":138558,"apply.run_cpu_time_us":4538,"apply.run_wall_time_us":295613,"num_ops":4,"prepare.queue_time_us":361716,"prepare.run_cpu_time_us":3617,"prepare.run_wall_time_us":139388,"replication_time_us":311369,"thread_start_us":332369,"threads_started":2}]]}
W20250114 08:58:52.183022 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 46304.8us
W20250114 08:58:52.384418 21547 consensus_meta.cc:287] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.658s user 0.009s sys 0.000s
I20250114 08:58:52.403934 21547 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:52.226284 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.406642 21550 raft_consensus.cc:2388] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:52.406205 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.408092 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:52.388381 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 264791us
I20250114 08:58:52.388612 21622 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.409198 21550 raft_consensus.cc:2388] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:52.413671 21624 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:58:52.414125 21626 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.412158 21594 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:52.413678 21629 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: []
W20250114 08:58:52.418839 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 106717us
I20250114 08:58:52.421672 21667 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:52.419741 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.424638 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.423003 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.425391 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:52.425767 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.426666 21547 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:52.441556 21547 raft_consensus.cc:604] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:52.434656 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.434713 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.437037 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.440393 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.443146 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:52.444449 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:52.428530 21553 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:52.446707 21626 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:52.513252 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.513571 21553 raft_consensus.cc:1235] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.514189 21626 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:52.448719 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.515069 21553 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.515219 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.515527 21623 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:52.450527 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.452198 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.516309 21628 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.516640 21629 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:52.516916 21553 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.453645 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.454368 21551 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:52.517655 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.453307 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.463744 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.459761 21552 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:52.282920 22690 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:58:52.260597 22715 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:52.236016 22723 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.554356 22690 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:52.554668 21548 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.295727 22768 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:52.138849 22772 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.476526 22808 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:52.264838 22818 raft_consensus.cc:695] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:52.295992 22833 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.262436 22835 raft_consensus.cc:695] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.477576 22837 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.566841 22835 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:52.569144 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 134636us
I20250114 08:58:52.569434 22837 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.265762 22848 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:52.571905 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:52.489542 22855 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.779s user 0.012s sys 0.006s
I20250114 08:58:52.277582 22857 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:52.289659 22861 consensus_meta.cc:287] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.635s user 0.006s sys 0.002s
I20250114 08:58:52.577867 22977 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:52.578404 22977 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:52.578882 22977 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.578758 22861 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.580116 21552 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.494455 22872 raft_consensus.cc:3049] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:52.580853 22872 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.580938 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1050) took 1042 ms. Trace:
I20250114 08:58:52.581267 21555 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:52.557044 22768 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.330824 22891 raft_consensus.cc:3049] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:52.277817 22896 raft_consensus.cc:3049] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:52.582780 22891 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.582953 22896 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.584156 22891 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.584399 22896 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.585834 22896 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.586280 22891 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.586429 21699 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.272836 22885 raft_consensus.cc:695] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:52.559707 22818 consensus_queue.cc:237] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.589733 22885 consensus_queue.cc:237] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.320426 22919 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20250114 08:58:52.272912 22920 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:52.602416 22976 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:52.581462 21552 rpcz_store.cc:276] 0114 08:58:51.538119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.579428 (+1041309us) service_pool.cc:225] Handling call
0114 08:58:52.579868 (+ 440us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.580615 (+ 747us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.580875 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:58:52.291245 22921 consensus_meta.cc:287] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.190s user 0.006s sys 0.007s
I20250114 08:58:52.506239 22923 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:52.287199 22925 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.304850 22899 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:52.610054 22925 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.276346 22943 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.315728 22965 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.613023 22965 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.338235 22967 raft_consensus.cc:3049] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:52.614114 22967 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.232404 22969 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.269205 22970 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.618676 22970 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.312389 22972 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:52.622804 22972 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:52.610463 22899 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:52.623323 22972 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.623814 22899 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.298364 22973 raft_consensus.cc:695] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.626142 22973 consensus_queue.cc:237] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.516788 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.629786 21623 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.635550 21551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "57f9cc1dd3c64647848a33a65b08c893"
from {username='slave'} at 127.0.0.1:52970
W20250114 08:58:52.539625 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 118487us
I20250114 08:58:52.547675 21626 raft_consensus.cc:2388] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:52.552562 21621 raft_consensus.cc:2388] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:52.562707 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.483974 22823 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:52.563493 21624 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.576433 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.574070 22975 consensus_queue.cc:1046] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 08:58:52.493016 22864 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:52.582113 21703 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:52.504978 22878 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:58:52.588369 21699 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.560942 21546 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.558204 22808 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.687889 22808 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:52.609278 22921 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.689246 22864 raft_consensus.cc:695] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.690580 22864 consensus_queue.cc:237] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.270921 22914 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:58:52.692557 22914 raft_consensus.cc:695] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:52.693693 22914 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.615511 22967 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.696300 22967 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.608330 21628 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
W20250114 08:58:52.700142 22963 consensus_meta.cc:287] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.577s user 0.010s sys 0.005s
I20250114 08:58:52.700891 21548 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:52.702868 21548 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.703559 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1052) took 1153 ms. Trace:
I20250114 08:58:52.610460 22861 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.638774 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1036) took 1334 ms. Trace:
I20250114 08:58:52.639153 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43436 (request call id 70) took 1017 ms. Trace:
I20250114 08:58:52.641343 22970 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:52.641880 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 136974us
I20250114 08:58:52.642980 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1052) took 1335 ms. Trace:
I20250114 08:58:52.603677 22920 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.649062 22715 raft_consensus.cc:3049] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:52.710650 21627 rpcz_store.cc:276] 0114 08:58:51.622006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:51.622424 (+ 418us) service_pool.cc:225] Handling call
0114 08:58:52.638757 (+1016333us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54000c8d54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:52.638949 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2231845760}
I20250114 08:58:52.721696 22715 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:52.722892 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.723191 22715 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.710271 21554 rpcz_store.cc:276] 0114 08:58:51.304008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.018318 (+714310us) service_pool.cc:225] Handling call
0114 08:58:52.018651 (+ 333us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:52.018979 (+ 328us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:52.019007 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:52.023781 (+ 4774us) log.cc:844] Serialized 26 byte log entry
0114 08:58:52.242642 (+218861us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:52.550140 (+307498us) log.cc:844] Serialized 14 byte log entry
0114 08:58:52.638203 (+ 88063us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.638326 (+ 123us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:52.638381 (+ 55us) raft_consensus.cc:1680] finished
0114 08:58:52.638392 (+ 11us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:52.638581 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5649,"prepare.queue_time_us":423898,"raft.queue_time_us":19256,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":10558,"thread_start_us":343607,"threads_started":2,"wal-append.queue_time_us":290605}
I20250114 08:58:52.727515 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 57) took 1171 ms. Trace:
I20250114 08:58:52.729413 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1268) took 1029 ms. Trace:
I20250114 08:58:52.729607 22715 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.729890 21703 rpcz_store.cc:276] 0114 08:58:51.699596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:51.699782 (+ 186us) service_pool.cc:225] Handling call
0114 08:58:51.700158 (+ 376us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:52.676757 (+976599us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.729023 (+ 52266us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b54001eebd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:52.729325 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2027234560}
I20250114 08:58:52.728013 21554 rpcz_store.cc:276] 0114 08:58:51.556195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.726257 (+1170062us) service_pool.cc:225] Handling call
0114 08:58:52.726784 (+ 527us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.727037 (+ 253us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:52.727073 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:52.727150 (+ 77us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:58:52.727190 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.727255 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:52.727392 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.734799 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 58) took 1178 ms. Trace:
I20250114 08:58:52.556407 22690 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.654850 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1067) took 1112 ms. Trace:
I20250114 08:58:52.655519 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1037) took 1264 ms. Trace:
W20250114 08:58:52.657970 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 86240.3us
W20250114 08:58:52.659078 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 329696us
I20250114 08:58:52.473541 22673 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:52.711298 22921 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.669978 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.739032 22690 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:52.739336 21621 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.736665 21547 rpcz_store.cc:276] 0114 08:58:51.542248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.653291 (+1111043us) service_pool.cc:225] Handling call
0114 08:58:52.653796 (+ 505us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.654319 (+ 523us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:52.654364 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:52.654436 (+ 72us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:58:52.654482 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.654575 (+ 93us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:52.654736 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.741920 21547 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.742597 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 59) took 1174 ms. Trace:
I20250114 08:58:52.742937 21547 rpcz_store.cc:276] 0114 08:58:51.568616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.741237 (+1172621us) service_pool.cc:225] Handling call
0114 08:58:52.741702 (+ 465us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.742296 (+ 594us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.742523 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.670326 21622 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:52.745342 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.746762 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.747114 21594 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:52.601130 22855 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.671396 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.748876 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.749814 21626 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:52.750545 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.674973 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:52.751546 21629 raft_consensus.cc:2388] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:52.751905 21621 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:52.752485 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.662510 22823 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:52.753582 22823 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.753991 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:52.744365 21547 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.683205 22878 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.701290 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:52.717437 21555 rpcz_store.cc:276] 0114 08:58:51.307536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.020522 (+712986us) service_pool.cc:225] Handling call
0114 08:58:52.035008 (+ 14486us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.581837 (+546829us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.642575 (+ 60738us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400116cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:52.642890 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239134464}
I20250114 08:58:52.738015 22673 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.747689 21622 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:52.670694 21553 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.748927 22974 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:52.751435 22855 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.752642 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:52.684659 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1048) took 1166 ms. Trace:
I20250114 08:58:52.605094 22923 raft_consensus.cc:695] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.755091 21627 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.735366 21554 rpcz_store.cc:276] 0114 08:58:51.556605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.733614 (+1177009us) service_pool.cc:225] Handling call
0114 08:58:52.734092 (+ 478us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.734389 (+ 297us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:52.734428 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:52.734481 (+ 53us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:58:52.734522 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.734588 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:52.734731 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.703977 21548 rpcz_store.cc:276] 0114 08:58:51.550511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.702125 (+1151614us) service_pool.cc:225] Handling call
0114 08:58:52.702608 (+ 483us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.703307 (+ 699us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.703493 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.758955 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1049) took 1240 ms. Trace:
I20250114 08:58:52.764230 21553 rpcz_store.cc:276] 0114 08:58:51.518448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.597892 (+1079444us) service_pool.cc:225] Handling call
0114 08:58:52.660843 (+ 62951us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.758712 (+ 97869us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.758902 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.768368 21669 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.758219 21624 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:52.701020 22963 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.736742 21549 rpcz_store.cc:276] 0114 08:58:51.344707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.106802 (+762095us) service_pool.cc:225] Handling call
0114 08:58:52.118838 (+ 12036us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:52.119192 (+ 354us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:52.119220 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:52.119674 (+ 454us) log.cc:844] Serialized 26 byte log entry
0114 08:58:52.385718 (+266044us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:52.547329 (+161611us) log.cc:844] Serialized 14 byte log entry
0114 08:58:52.595442 (+ 48113us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.597664 (+ 2222us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:52.600026 (+ 2362us) raft_consensus.cc:1680] finished
0114 08:58:52.600045 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:52.608755 (+ 8710us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3228,"prepare.queue_time_us":367885,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":34085,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":17039,"thread_start_us":422889,"threads_started":2,"wal-append.queue_time_us":227315}
I20250114 08:58:52.782851 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.787266 21621 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.641391 22831 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.790290 21548 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:52.703989 22833 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.776667 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1055) took 1188 ms. Trace:
I20250114 08:58:52.726807 22881 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.766548 22972 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.801460 21555 raft_consensus.cc:1235] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:52.798272 21554 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250114 08:58:52.802098 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:52.763206 21546 rpcz_store.cc:276] 0114 08:58:51.518151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.560134 (+1041983us) service_pool.cc:225] Handling call
0114 08:58:52.560676 (+ 542us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.684220 (+123544us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.684565 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.791945 22974 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.807889 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 64) took 1200 ms. Trace:
I20250114 08:58:52.809964 21555 rpcz_store.cc:276] 0114 08:58:51.606342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.774672 (+1168330us) service_pool.cc:225] Handling call
0114 08:58:52.789572 (+ 14900us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.805431 (+ 15859us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.806397 (+ 966us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.800727 22861 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:52.816532 21552 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:52.819223 21553 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:58:52.816071 22712 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20250114 08:58:52.820314 22712 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:52.820689 22712 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:52.821084 22673 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:52.821627 22712 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.823078 22712 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.824330 22883 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
W20250114 08:58:52.824535 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 155263us
I20250114 08:58:52.825896 22883 raft_consensus.cc:3049] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:52.826282 22872 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.827663 22872 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:52.827531 22673 consensus_queue.cc:1046] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:52.819257 21630 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:52.828860 22891 raft_consensus.cc:3049] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:52.829325 22891 raft_consensus.cc:738] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.829731 21518 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:52.830034 21623 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:52.830339 22891 consensus_queue.cc:260] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.831105 22899 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.796371 21547 rpcz_store.cc:276] 0114 08:58:51.580115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.743723 (+1163608us) service_pool.cc:225] Handling call
0114 08:58:52.744179 (+ 456us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.754895 (+ 10716us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.768822 (+ 13927us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:52.795845 22768 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.833030 22920 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:52.815596 22861 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:52.826292 22883 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.835369 22673 consensus_queue.cc:1046] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:52.836036 22994 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:52.837175 22848 consensus_queue.cc:1046] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:52.847826 21627 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:52.832162 22977 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.865126 22891 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.865244 21669 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.866274 22899 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.866768 22977 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.867128 22848 raft_consensus.cc:3049] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:52.867513 22848 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:52.868446 22848 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.869939 22848 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.871012 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:52.871678 21699 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.862958 22861 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:52.875537 22975 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:52.875970 22975 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:52.876541 21554 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 08:58:52.852007 23002 log.cc:926] Time spent T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20250114 08:58:52.848778 22878 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.856577 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:52.856755 22673 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:52.882740 22673 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.883777 22673 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.885352 22673 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.856992 22923 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.887928 22768 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:52.888115 22269 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:52.895705 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:52.896301 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:52.896729 21700 raft_consensus.cc:2388] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:52.897075 21698 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.901306 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1069) took 1358 ms. Trace:
I20250114 08:58:52.901700 22832 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:58:52.903127 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:52.903254 22833 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.903304 22832 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:52.859400 22831 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:52.860342 22974 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:52.901716 21552 rpcz_store.cc:276] 0114 08:58:51.542948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.655851 (+1112903us) service_pool.cc:225] Handling call
0114 08:58:52.656343 (+ 492us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:52.816993 (+160650us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:52.901076 (+ 84083us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540028f0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:52.901247 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":359142144}
I20250114 08:58:52.860759 22843 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:52.876555 22975 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:52.855371 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 213076us
I20250114 08:58:52.902132 21703 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:52.859156 22883 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.903848 21700 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:52.942922 21669 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:52.939054 22240 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:52.961097 22963 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:52.961956 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet 3b369b35736c413ca069db061ed36bc5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250114 08:58:52.962698 22823 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.880553 22885 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:58:52.965500 21698 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:52.974480 22883 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:52.974987 22987 log.cc:926] Time spent T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
W20250114 08:58:52.974512 23003 log.cc:926] Time spent T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250114 08:58:52.977190 22881 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:52.978114 22963 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:52.983412 21521 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:52.984369 22885 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:52.984767 22885 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:52.985136 22885 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.985447 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:52.986183 21695 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.968185 22843 consensus_queue.cc:1046] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20250114 08:58:52.988788 22843 raft_consensus.cc:3049] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:52.989251 22843 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:52.990283 22843 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:52.991492 22967 consensus_queue.cc:1046] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:52.992147 22963 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:52.992389 21700 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:52.878180 22864 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20250114 08:58:52.993445 22972 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:52.954064 22891 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250114 08:58:52.994119 22768 consensus_queue.cc:1046] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:52.997859 21695 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:52.998169 22808 consensus_queue.cc:1046] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:53.001118 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:53.005821 21550 raft_consensus.cc:1270] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:53.008316 21521 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:53.017261 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.020877 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:53.023416 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 142941us
I20250114 08:58:53.028024 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.123404 22823 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:53.163286 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 54573.1us
I20250114 08:58:53.211560 21627 raft_consensus.cc:2405] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:53.022367 22885 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:53.033385 22995 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:53.204411 21699 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:53.034485 22271 raft_consensus.cc:2405] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250114 08:58:53.209450 21622 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:52.968318 22861 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.221019 22843 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.222121 21703 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:58:53.219018 21668 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:53.208365 22896 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.622s user 0.010s sys 0.005s
I20250114 08:58:53.203198 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.231308 21669 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:53.233927 22837 raft_consensus.cc:3049] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:53.234427 22837 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:53.235553 22837 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.221750 22861 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:53.250638 21549 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:53.252991 22837 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:53.253155 22723 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250114 08:58:53.253214 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.241302 22974 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.256795 22896 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:53.256902 22974 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:53.257669 21668 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:53.257905 21550 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.259954 22974 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:53.261323 21555 raft_consensus.cc:1270] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:53.262250 22993 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:53.262656 22993 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:53.263010 22993 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:53.264609 22963 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.267273 21552 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.269673 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.270388 22896 raft_consensus.cc:695] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.270481 22990 raft_consensus.cc:3049] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:53.270926 22990 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.271437 22896 consensus_queue.cc:237] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.272918 22972 raft_consensus.cc:3049] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
W20250114 08:58:53.273212 23001 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250114 08:58:53.273479 22972 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.274924 22768 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:53.274529 22972 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.275170 21547 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:53.248277 22808 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:53.276299 22823 consensus_queue.cc:1046] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:53.276888 22808 raft_consensus.cc:695] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:53.277666 22977 raft_consensus.cc:3049] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:53.277612 22823 consensus_queue.cc:1046] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:53.278250 22977 raft_consensus.cc:738] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.277907 22808 consensus_queue.cc:237] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.279224 22823 raft_consensus.cc:3049] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:53.279122 22977 consensus_queue.cc:260] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.279657 22823 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.280427 22977 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:53.281742 22768 raft_consensus.cc:695] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.282349 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.283257 21553 raft_consensus.cc:2405] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:53.282881 22768 consensus_queue.cc:237] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
W20250114 08:58:53.283795 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 60109.7us
I20250114 08:58:53.287181 22972 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.291850 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.288152 22878 raft_consensus.cc:3049] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:53.292536 22878 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.293315 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.293640 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.293516 22878 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.294440 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.295644 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.295850 22848 raft_consensus.cc:988] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:58:53.280617 22823 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.290024 22891 consensus_queue.cc:1046] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:53.271962 22990 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.296006 22878 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:53.300552 21669 leader_election.cc:400] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:53.296197 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:53.300886 21669 leader_election.cc:403] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:53.301465 21547 raft_consensus.cc:2388] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:53.301494 22861 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:53.301772 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:53.297966 22990 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:53.297277 22891 raft_consensus.cc:3049] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:53.297168 22845 raft_consensus.cc:1076] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Signalling peer d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:58:53.302696 22861 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:53.303130 22891 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.303326 22861 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:53.303787 22861 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:53.304172 22891 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.305819 22891 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.309582 22823 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.314380 22861 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.323144 22993 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.329237 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:53.333173 23008 log.cc:926] Time spent T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250114 08:58:53.338871 22271 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250114 08:58:53.333242 21451 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 6, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 6 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:58:53.340436 21551 raft_consensus.cc:604] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:53.340956 21551 raft_consensus.cc:612] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:58:53.340706 22977 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20250114 08:58:53.345089 22872 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.517s user 0.013s sys 0.000s
I20250114 08:58:53.356922 21622 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:53.361017 21548 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:53.368937 21553 raft_consensus.cc:2363] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:53.374919 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:53.375833 21552 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:53.376271 21548 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:53.379850 21550 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250114 08:58:53.384932 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 529573us
I20250114 08:58:53.385565 21547 raft_consensus.cc:1270] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:53.385381 22975 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.386910 21702 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.386879 22943 consensus_queue.cc:1035] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250114 08:58:53.387864 22975 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:53.360908 22864 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20250114 08:58:53.414002 21550 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250114 08:58:53.417353 22993 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:53.422792 22969 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:53.424584 22864 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 08:58:53.425638 22673 consensus_meta.cc:287] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.540s user 0.014s sys 0.009s
I20250114 08:58:53.426503 21548 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:53.418236 22843 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20250114 08:58:53.429172 22885 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:53.430096 22843 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:53.379751 22832 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250114 08:58:53.432454 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:53.419826 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
W20250114 08:58:53.460152 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 451883us
I20250114 08:58:53.462558 22269 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:53.463063 22266 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250114 08:58:53.463194 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 47146.7us
I20250114 08:58:53.464025 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:53.465946 22274 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:53.467070 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:53.467314 22848 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:53.468323 22673 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:58:53.468933 22673 raft_consensus.cc:695] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.468971 21703 raft_consensus.cc:1270] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:53.497840 21703 raft_consensus.cc:1235] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.505801 21699 raft_consensus.cc:1235] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.469997 22673 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:53.476135 22861 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:53.512318 21520 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:53.497030 22690 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250114 08:58:53.526258 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 113752us
I20250114 08:58:53.525672 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.533080 21703 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.529002 22837 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250114 08:58:53.548255 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 180124us
I20250114 08:58:53.549490 21669 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:53.552428 21554 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
W20250114 08:58:53.548326 23009 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.086s user 0.000s sys 0.004s
I20250114 08:58:53.531589 21695 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:53.573462 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 63811.5us
I20250114 08:58:53.599140 21520 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:53.607327 21622 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:53.601444 21702 raft_consensus.cc:1235] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.576609 22864 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20250114 08:58:53.574942 22969 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20250114 08:58:53.607010 22861 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:53.619386 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.620332 21622 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:53.622596 22818 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:53.623401 21555 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250114 08:58:53.623232 22818 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.623683 21549 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250114 08:58:53.624583 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 88300.7us
I20250114 08:58:53.625418 21521 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:53.616689 22768 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.633519 21703 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:53.636152 22270 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:58:53.641506 22837 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:53.637436 23014 consensus_queue.cc:1046] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250114 08:58:53.635295 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.636348 21520 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:53.645879 21703 raft_consensus.cc:2405] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:53.659205 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:53.660090 22920 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:53.660818 21703 raft_consensus.cc:2405] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:53.659515 21521 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:53.633205 22861 raft_consensus.cc:695] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:53.660708 22920 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:53.637154 22271 raft_consensus.cc:1270] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20250114 08:58:53.649513 21593 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:53.644404 23014 raft_consensus.cc:3049] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:53.679915 22690 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:53.653357 22896 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:53.671329 22831 consensus_meta.cc:287] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.718s user 0.000s sys 0.013s
W20250114 08:58:53.684392 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 96772.1us
I20250114 08:58:53.701642 21630 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.702414 21624 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.702922 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.703670 21629 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:53.704282 21625 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:53.704490 21626 raft_consensus.cc:2405] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20250114 08:58:53.705844 23030 log.cc:926] Time spent T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20250114 08:58:53.706197 22864 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.706741 22864 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:53.707387 22969 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:58:53.695943 22920 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.708792 22872 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.709275 22872 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.709661 21628 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:53.689343 21627 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.681106 22768 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:53.724148 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.725013 21624 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:53.725154 22864 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:53.725867 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.726612 21624 raft_consensus.cc:2363] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:53.726536 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:53.691403 22832 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250114 08:58:53.692838 23025 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.056s user 0.001s sys 0.001s
W20250114 08:58:53.693431 22241 leader_election.cc:336] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.021s (SENT)
W20250114 08:58:53.695714 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 59485.4us
I20250114 08:58:53.697446 22818 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:53.698086 22896 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.679332 23014 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:53.711256 21621 raft_consensus.cc:1235] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:53.694386 22861 consensus_queue.cc:237] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.677954 22271 pending_rounds.cc:77] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Aborting all ops after (but not including) 4
I20250114 08:58:53.722783 22872 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:53.727442 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.728163 22808 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:53.777484 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:53.854939 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.855685 21695 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:53.856681 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.858207 21696 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:53.859304 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.859416 22808 raft_consensus.cc:695] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:53.860646 22843 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.860503 22808 consensus_queue.cc:237] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.860960 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.861186 22843 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:53.861816 21701 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:53.862666 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:53.863226 22843 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:53.863402 21701 raft_consensus.cc:2363] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:53.866681 22768 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:53.867314 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:53.778586 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 50887.2us
I20250114 08:58:53.856287 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.864240 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.727278 21698 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:53.777674 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.812850 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:53.876940 21629 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:58:53.879001 21703 raft_consensus.cc:2388] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:53.877601 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.729067 21630 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:53.880230 21628 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:53.788492 22848 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.852684 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:53.888403 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 254477us
I20250114 08:58:53.889297 21697 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.890360 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.890427 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.906606 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:53.897790 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.909703 21699 raft_consensus.cc:2405] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:53.911435 21622 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250114 08:58:53.912063 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.684104 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.874687 22271 pending_rounds.cc:99] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250114 08:58:53.914273 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:53.917693 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.918224 21624 raft_consensus.cc:2405] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 08:58:53.915611 21596 leader_election.cc:336] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.541s (SENT)
I20250114 08:58:53.918604 21703 raft_consensus.cc:2405] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:53.873656 22832 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:53.924830 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1175) took 1420 ms. Trace:
I20250114 08:58:53.929066 21626 rpcz_store.cc:276] 0114 08:58:52.492033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:52.750303 (+258270us) service_pool.cc:225] Handling call
0114 08:58:53.905635 (+1155332us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b54001522d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:53.912055 (+ 6420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2191837952}
I20250114 08:58:53.973062 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:53.973940 23014 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:53.788028 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:53.976543 23019 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20250114 08:58:53.896157 22864 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.976964 21623 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:53.922817 21701 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:53.955883 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:54.003578 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:54.011547 21696 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:53.894832 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:54.009465 21628 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:54.023864 21630 raft_consensus.cc:2388] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:53.916970 22831 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.813581 22837 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.168s
I20250114 08:58:53.922209 22271 raft_consensus.cc:2883] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:58:53.999166 22768 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.031605 22269 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:54.042688 22768 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.043542 21629 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:54.018615 21695 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:54.045873 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:54.019351 22843 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.055481 22843 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.036506 22831 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.058547 22843 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:53.982260 22832 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:54.059216 21627 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:54.061160 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 145600us
W20250114 08:58:54.061825 21596 leader_election.cc:336] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.548s (SENT)
I20250114 08:58:54.062181 21596 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:54.063462 21596 leader_election.cc:336] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.603s (SENT)
I20250114 08:58:54.063818 21596 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:54.065469 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 372081us
I20250114 08:58:54.065863 23012 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:53.903640 22818 consensus_queue.cc:1046] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.119s
I20250114 08:58:54.003772 22974 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.014346 22833 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.021514 21520 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:54.058641 22831 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:54.035733 23023 log.cc:926] Time spent T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20250114 08:58:53.913249 22848 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.131217 22977 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.131719 22977 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.132587 22974 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.127801 21697 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:54.133749 22977 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.134618 22974 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.076242 21704 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:54.134632 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.068336 23014 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:54.128616 21627 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:54.137140 21701 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:54.141139 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.142107 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.025907 22864 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.077749 22843 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.159555 21701 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:54.128134 22872 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.162858 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.165903 21700 raft_consensus.cc:2388] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:54.164538 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.168908 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.169431 22271 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 08:58:54.169639 21701 raft_consensus.cc:2363] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:54.170447 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.171622 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.139058 22974 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:54.172232 21697 raft_consensus.cc:2363] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:54.172345 21701 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:54.171279 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:54.181234 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 263579us
I20250114 08:58:54.181813 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.167202 22848 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.136353 23012 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:54.185858 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.187315 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.190178 21697 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:54.192732 21701 raft_consensus.cc:2363] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3.
I20250114 08:58:54.194671 21625 raft_consensus.cc:1212] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20250114 08:58:54.201268 21625 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:54.206353 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.207015 21701 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:54.208118 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.165795 22843 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.210573 21697 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:54.191864 22974 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:54.173307 22872 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.183007 22993 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20250114 08:58:54.210491 23039 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250114 08:58:54.217020 21701 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20250114 08:58:54.211135 22241 leader_election.cc:336] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.231s (SENT)
I20250114 08:58:54.221686 21703 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:54.220599 21701 pending_rounds.cc:77] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:58:54.230778 21701 pending_rounds.cc:99] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:58:54.231150 21701 raft_consensus.cc:2883] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:58:54.183614 22833 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.244076 21697 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:54.241585 22843 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.245858 22864 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.246173 21703 raft_consensus.cc:1270] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:54.248230 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.254801 21703 raft_consensus.cc:1270] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250114 08:58:54.246234 22241 leader_election.cc:336] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.235s (SENT)
I20250114 08:58:54.240907 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.267143 21697 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:54.276017 21704 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:54.277607 21703 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 08:58:54.277954 21701 raft_consensus.cc:1270] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:54.248492 22963 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.284653 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.278020 21703 pending_rounds.cc:77] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:58:54.286072 21703 pending_rounds.cc:99] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:58:54.286427 21703 raft_consensus.cc:2883] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:58:54.252625 22864 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.290959 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:54.293790 21696 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:54.295063 22864 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:54.295818 23043 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250114 08:58:54.296970 21623 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:54.297333 22864 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.297981 21621 raft_consensus.cc:1235] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
W20250114 08:58:54.299079 23044 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.128s user 0.000s sys 0.002s
I20250114 08:58:54.252965 22974 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20250114 08:58:54.310170 21695 raft_consensus.cc:1235] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:54.313555 21593 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:54.314654 22843 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:54.315158 22843 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:54.315414 22872 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.315590 22843 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:54.316882 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 70697.6us
I20250114 08:58:54.319047 21701 raft_consensus.cc:1235] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:54.219283 22832 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:54.323650 21697 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:58:54.332396 21700 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
W20250114 08:58:54.337464 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 274016us
I20250114 08:58:54.337122 23050 consensus_queue.cc:1035] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250114 08:58:54.291967 22963 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.346139 23046 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20250114 08:58:54.371941 23053 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.372241 22818 raft_consensus.cc:3049] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:54.372488 22833 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.372668 22818 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:58:54.353894 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 442688us
I20250114 08:58:54.354370 22925 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250114 08:58:54.373626 22818 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.374926 22818 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:54.375227 22925 consensus_queue.cc:1046] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:54.351316 22832 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:54.376268 23014 consensus_queue.cc:1046] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:54.376837 22920 raft_consensus.cc:3049] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:54.377305 22920 raft_consensus.cc:738] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:54.378140 23014 raft_consensus.cc:3049] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:54.379212 23014 raft_consensus.cc:738] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:54.381204 23014 consensus_queue.cc:260] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.380129 22848 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:54.376489 22925 consensus_queue.cc:1046] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:54.392302 22848 raft_consensus.cc:3049] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:54.392691 22848 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:58:54.379583 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 462301us
I20250114 08:58:54.393780 21520 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:54.393568 22848 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.394572 22848 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:54.395121 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:54.396450 22808 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:54.397182 22808 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:54.397632 22808 raft_consensus.cc:695] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:54.398530 22808 consensus_queue.cc:237] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.378219 22920 consensus_queue.cc:260] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.362308 23050 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.363534 21623 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:55318
I20250114 08:58:54.368643 22896 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.372536 23053 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.375272 21521 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:54.360616 22885 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.346290 21699 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:54.361843 22963 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.366603 23012 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:54.365072 21596 leader_election.cc:336] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.644s (SENT)
I20250114 08:58:54.390316 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:54.390569 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:54.428143 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 233958us
I20250114 08:58:54.383867 22837 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:54.471397 23014 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:54.474713 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 99513.2us
I20250114 08:58:54.355827 22768 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.484365 22896 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.490566 21596 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:54.491402 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 95136.1us
I20250114 08:58:54.474295 22872 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.361595 22843 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:54.499492 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 108995us
I20250114 08:58:54.498137 21699 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:54.498574 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:54.547106 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 46680.7us
I20250114 08:58:54.497828 22994 consensus_queue.cc:1046] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20250114 08:58:54.544852 22843 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:54.557112 22896 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:54.545229 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 180198us
I20250114 08:58:54.545074 21520 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:54.562543 21669 leader_election.cc:400] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:54.565789 21669 leader_election.cc:403] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:54.545495 21704 raft_consensus.cc:1270] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:58:54.481453 22925 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:54.480955 21623 raft_consensus.cc:604] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:54.494161 22963 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:54.564815 21696 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:54.475097 23050 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.526261 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:54.498692 23053 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.487215 22837 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:54.544624 23046 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:54.619171 23050 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.531133 22920 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:54.547341 23054 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:54.606365 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 113214us
I20250114 08:58:54.602720 22690 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.542766 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:54.599610 22673 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:58:54.485710 22885 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
W20250114 08:58:54.635864 21668 leader_election.cc:336] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.640s (SENT)
I20250114 08:58:54.545367 22872 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.617918 22837 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.612210 22994 raft_consensus.cc:3049] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:54.546298 23012 consensus_queue.cc:1046] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20250114 08:58:54.618913 23046 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:54.544160 22989 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20250114 08:58:54.566962 22891 consensus_queue.cc:1046] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20250114 08:58:54.627938 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:58:54.639264 22881 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:54.716845 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 79905us
W20250114 08:58:54.669247 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 320732us
I20250114 08:58:54.673322 22994 raft_consensus.cc:738] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:54.649358 22690 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.717996 22271 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:54.720191 22993 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.759663 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:54.722002 22861 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.723187 22837 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:54.765357 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 203511us
I20250114 08:58:54.769728 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:54.749410 22848 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:54.720938 22881 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:54.793053 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:54.801932 21696 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:54.818487 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:54.821435 21669 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:54.797080 22994 consensus_queue.cc:260] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.763934 22993 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.849349 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 71) took 1557 ms. Trace:
I20250114 08:58:54.813946 22768 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:58:54.772848 22861 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.838480 22977 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:54.837823 22891 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:54.879943 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 206824us
I20250114 08:58:54.840652 22690 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.837697 22896 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.862605 23063 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.813437 22989 consensus_queue.cc:1046] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20250114 08:58:54.875726 22925 raft_consensus.cc:695] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:58:54.911572 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 259435us
I20250114 08:58:54.902405 22885 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:54.936976 23060 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20250114 08:58:54.938699 22989 raft_consensus.cc:3049] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:54.939155 22989 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:54.940943 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:54.948899 23046 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:54.950114 22818 consensus_meta.cc:287] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.575s user 0.007s sys 0.004s
W20250114 08:58:54.950400 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 114099us
I20250114 08:58:54.951083 22861 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.951478 22241 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:54.952234 22925 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:54.953207 22977 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:54.953934 22977 raft_consensus.cc:695] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:54.954502 22673 raft_consensus.cc:3049] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:54.954931 22673 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:54.955037 22977 consensus_queue.cc:237] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.957463 22993 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.958135 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:54.958822 21548 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:54.906955 23070 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.962990 22843 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.964782 23070 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.964884 22843 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.963197 23071 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:54.963667 22881 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:54.922412 23073 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.967309 23070 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.912560 22994 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:54.967582 23073 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:54.969090 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:54.955826 22673 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.970252 23073 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:54.964035 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:54.933522 23050 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.970795 22673 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:54.921670 22848 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:54.903425 22891 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:54.983961 21596 leader_election.cc:336] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.706s (SENT)
I20250114 08:58:54.984627 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:54.914638 22872 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:54.972322 22885 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.992353 21621 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:54.993342 21628 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:54.993647 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:54.993896 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.001219 21629 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:54.995700 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.002465 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:54.972306 22843 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:54.857746 21451 rpcz_store.cc:276] 0114 08:58:53.291906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:53.292061 (+ 155us) service_pool.cc:225] Handling call
0114 08:58:54.849253 (+1557192us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:53.556934 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:53.557717 (+ 783us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:53.557774 (+ 57us) write_op.cc:435] Acquired schema lock
0114 08:58:53.557808 (+ 34us) tablet.cc:659] Decoding operations
0114 08:58:53.558341 (+ 533us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:53.558446 (+ 105us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:53.558475 (+ 29us) tablet.cc:682] Acquiring locks for 1 operations
0114 08:58:53.558839 (+ 364us) tablet.cc:698] Row locks acquired
0114 08:58:53.558873 (+ 34us) write_op.cc:260] PREPARE: finished
0114 08:58:53.559053 (+ 180us) write_op.cc:270] Start()
0114 08:58:53.559204 (+ 151us) write_op.cc:276] Timestamp: P: 1736845133558984 usec, L: 0
0114 08:58:53.559235 (+ 31us) op_driver.cc:348] REPLICATION: starting
0114 08:58:53.559965 (+ 730us) log.cc:844] Serialized 538 byte log entry
0114 08:58:54.298691 (+738726us) op_driver.cc:464] REPLICATION: finished
0114 08:58:54.800573 (+501882us) write_op.cc:301] APPLY: starting
0114 08:58:54.800696 (+ 123us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:54.810116 (+ 9420us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:54.840000 (+ 29884us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:54.840339 (+ 339us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:54.840603 (+ 264us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 08:58:54.840663 (+ 60us) write_op.cc:312] APPLY: finished
0114 08:58:54.840959 (+ 296us) log.cc:844] Serialized 22 byte log entry
0114 08:58:54.841152 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:54.841238 (+ 86us) write_op.cc:454] Released schema lock
0114 08:58:54.841295 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:54.841667 (+ 372us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:54.841717 (+ 50us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":461315,"apply.run_cpu_time_us":2539,"apply.run_wall_time_us":62551,"num_ops":1,"prepare.queue_time_us":156748,"prepare.run_cpu_time_us":13936,"prepare.run_wall_time_us":254245,"raft.queue_time_us":52951,"raft.run_cpu_time_us":32067,"raft.run_wall_time_us":339906,"replication_time_us":739305,"thread_start_us":617724,"threads_started":4,"wal-append.queue_time_us":576928}]]}
I20250114 08:58:54.977651 23046 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:55.009649 21624 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:55.010849 21622 raft_consensus.cc:1270] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:54.980158 22832 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:54.978260 22861 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:55.017534 21548 raft_consensus.cc:2405] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:55.019870 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.020757 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.021510 21626 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
W20250114 08:58:55.012863 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 449722us
I20250114 08:58:55.013087 21629 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:55.014643 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.025133 21623 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:54.999372 23054 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.000195 23070 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:54.913273 23063 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.045950 21628 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:55.025774 22843 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:54.940207 22989 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:55.006523 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 55163.4us
I20250114 08:58:54.982153 23050 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:54.983636 22848 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.058509 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.065340 21551 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
W20250114 08:58:55.097127 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 115196us
I20250114 08:58:55.097716 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.098613 21555 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:55.009122 22885 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.032301 22808 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:54.999251 22240 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:55.113061 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 323065us
I20250114 08:58:55.033558 22271 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:55.026338 22861 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.064049 22872 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:55.121713 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 376918us
I20250114 08:58:55.064332 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.116678 22808 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.023128 22832 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.135584 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:55.137147 21555 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:55.049118 23070 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.117010 23050 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.108933 22989 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.049186 23054 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:55.124961 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 303609us
W20250114 08:58:55.155472 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 45717.2us
I20250114 08:58:55.154121 21521 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:55.159319 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.166672 22993 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.167203 23070 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.167270 22993 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.170877 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.127148 23082 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:58:55.209908 21669 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:55.155709 22768 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:58:55.212205 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 164540us
W20250114 08:58:55.154948 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 161449us
I20250114 08:58:55.162314 22896 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:55.214650 21593 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.214383 21627 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:55.219094 23073 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:55.220887 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:55.223270 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 83916.5us
I20250114 08:58:55.224925 21669 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.126888 22872 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.176925 23075 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:55.235047 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.230553 21550 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:55.215799 22837 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:55.224855 22861 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:55.159864 23088 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20250114 08:58:55.176765 22993 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:55.212743 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 115819us
I20250114 08:58:55.254786 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.252979 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.234722 23070 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:55.175787 23089 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20250114 08:58:55.253399 22963 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.215906 22881 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250114 08:58:55.249795 23075 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.274055 21549 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:55.276777 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.250240 22891 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:55.282366 21520 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:55.284715 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.259477 22861 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:55.254076 23097 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:55.236352 23073 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.228811 23095 consensus_queue.cc:1046] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20250114 08:58:55.278452 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.315272 22872 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:55.315904 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.316632 21700 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:55.316983 23088 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.317754 22808 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.306102 23082 consensus_queue.cc:1046] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250114 08:58:55.318411 22832 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.318815 23082 raft_consensus.cc:3049] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:55.319201 23082 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:55.320138 23082 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:55.302811 22861 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:55.321424 23082 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.322089 22861 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.323544 23046 consensus_queue.cc:1046] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:58:55.282301 22837 raft_consensus.cc:695] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:55.233773 22896 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.309897 22885 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.290263 22963 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.312608 23063 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.312762 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:55.317884 23099 log.cc:926] Time spent T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20250114 08:58:55.309428 23087 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
W20250114 08:58:55.316110 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 79961us
I20250114 08:58:55.345862 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.346531 21704 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:55.327972 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:55.298415 22848 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.290407 23075 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.339574 23053 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.340575 23097 raft_consensus.cc:695] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:55.287041 22768 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.342203 22837 consensus_queue.cc:237] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.344144 23073 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.344595 23054 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.348752 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:55.357885 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 133051us
I20250114 08:58:55.377346 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.377743 21669 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.363867 21521 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:55.364656 21696 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 08:58:55.367810 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 114542us
I20250114 08:58:55.333536 22993 raft_consensus.cc:3049] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:55.325814 23070 raft_consensus.cc:695] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:55.366928 21698 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:55.383673 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.382269 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:55.386355 21596 leader_election.cc:400] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:55.386761 21596 leader_election.cc:403] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:58:55.382460 23095 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.384531 21553 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:55.391650 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.393891 21594 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.398114 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.375766 22848 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.404256 21697 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:55.405916 21594 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.383896 23073 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.407799 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.388999 22993 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:55.385401 22872 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:55.394846 23095 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.382898 23051 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:55.409024 21594 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.397681 23054 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:55.445232 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 65797.8us
I20250114 08:58:55.421684 21555 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:58:55.399612 23092 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:55.418628 23047 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.405529 23053 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:55.409333 22864 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:55.407289 22848 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.400590 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.446202 21596 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.446599 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.502979 21555 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
W20250114 08:58:55.447893 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 44423.7us
I20250114 08:58:55.396899 22891 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250114 08:58:55.507540 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.415238 23108 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:55.524653 23108 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.525229 23108 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.526840 22993 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.413085 23073 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:55.528292 22993 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.528743 23073 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.504810 21594 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.472687 23047 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.532999 23047 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.504853 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.534231 21552 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:55.535265 21552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "57f9cc1dd3c64647848a33a65b08c893"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:55.535970 21552 raft_consensus.cc:604] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Received request to transfer leadership to TS 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:55.537149 22831 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.537724 22831 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:55.538189 22831 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.541008 21594 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.542027 23101 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:55.542451 23101 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.542773 23101 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:55.545241 23096 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
W20250114 08:58:55.546394 23107 log.cc:926] Time spent T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20250114 08:58:55.536401 21552 raft_consensus.cc:612] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:58:55.549363 23070 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:55.551110 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20250114 08:58:55.424571 23095 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.470651 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.510283 23052 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.510895 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.512634 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:55.506090 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:55.473407 22832 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:55.448395 23054 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.551178 23101 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.556492 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:55.556694 22881 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:55.556886 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:55.559116 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 112993us
I20250114 08:58:55.580519 21596 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.422798 22673 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20250114 08:58:55.447284 23051 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.565768 21555 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:55.499251 22864 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:55.505116 22848 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.573215 23075 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:55.578053 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 185157us
I20250114 08:58:55.445092 22872 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:55.583920 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.581584 23061 raft_consensus.cc:3049] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:55.586225 21669 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.577291 22832 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.579478 23054 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.605746 21596 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.600965 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:55.584985 23051 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.585247 22896 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:55.585055 22963 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:58:55.610826 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 54066us
I20250114 08:58:55.586508 22864 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:55.611414 21669 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:55.676838 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 65482.4us
I20250114 08:58:55.677699 21669 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.668800 23101 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.621280 21596 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:55.603116 21668 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.186s (SENT) [suppressed 33 similar messages]
I20250114 08:58:55.604602 23052 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:55.638511 22843 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.610800 23083 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.609613 22832 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:55.617424 22891 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:55.668613 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 49683us
I20250114 08:58:55.669235 23075 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.505640 23097 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:55.670468 22872 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:55.718591 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.674456 23108 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.600059 22925 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:55.605508 23061 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
W20250114 08:58:55.702239 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 99176.1us
I20250114 08:58:55.703960 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.723037 21520 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:55.602734 23092 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20250114 08:58:55.727666 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:55.685851 22994 consensus_meta.cc:287] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.706s user 0.010s sys 0.002s
I20250114 08:58:55.708345 22891 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.739562 22891 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.739815 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.740444 21623 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:55.740273 23047 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.741516 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.742112 21623 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:55.742343 23047 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.743026 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.743738 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:55.708453 23111 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20250114 08:58:55.693853 22963 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:58:55.719677 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 98506.8us
I20250114 08:58:55.717626 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:55.719401 21625 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:55.758538 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.704993 22974 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.705468 23083 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.725780 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.722126 23052 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.724529 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.724617 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.723454 22872 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.731717 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:55.733191 21520 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:55.733824 23063 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:58:55.738191 22831 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.738643 22832 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.728487 21629 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:55.747287 23073 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:55.781473 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 48372.5us
I20250114 08:58:55.743639 21623 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:55.693018 22673 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20250114 08:58:55.761363 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:55.765509 21621 raft_consensus.cc:2388] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:55.761600 22963 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.765921 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:55.803573 21555 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.773504 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:55.715202 22993 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250114 08:58:55.781152 21625 raft_consensus.cc:1235] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:58:55.715457 22848 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.784057 21626 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:55.725997 21622 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:55.757280 21547 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:55.767918 22925 raft_consensus.cc:695] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:55.767963 23083 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.815678 21551 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:55.822839 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:55.823813 21596 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.802356 23014 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:55.808620 22963 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.761332 23051 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:55.818356 23047 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:55.767813 22974 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.846987 21596 leader_election.cc:400] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:55.851557 21596 leader_election.cc:403] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:55.780243 22872 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:55.847854 23101 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.864953 22943 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:55.867828 23051 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:58:55.870586 22848 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.870852 22872 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:55.871824 21624 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:55.872349 22831 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.873773 23073 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.864996 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.875254 21702 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:55.876118 22832 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:55.878739 22891 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.879977 23108 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:55.880311 23047 raft_consensus.cc:695] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:55.853113 23075 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:55.843950 23063 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:58:55.884977 23071 raft_consensus.cc:3049] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:55.884708 23054 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.838991 23061 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:55.874138 22925 consensus_queue.cc:237] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.897853 21596 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:55.897872 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:55.941404 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:55.903762 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.907944 21669 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.910275 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:55.943370 21622 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:55.897166 21627 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:55.914443 21555 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:55.944551 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.917977 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.918599 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.919191 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.922251 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.938221 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.942190 21550 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:55.946416 21701 raft_consensus.cc:2371] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3. Re-sending same reply.
I20250114 08:58:55.949467 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.948534 21594 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:55.949527 21704 raft_consensus.cc:2388] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:55.916479 22848 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.946902 21697 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:55.945297 21703 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:55.956336 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:55.957767 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 60036.6us
I20250114 08:58:55.958846 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:55.960438 21596 leader_election.cc:400] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:55.960733 21596 leader_election.cc:403] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:55.962272 21555 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:55.914767 22896 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.963761 23097 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:55.964211 23097 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:55.964149 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.964635 23051 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:55.965389 23051 raft_consensus.cc:695] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:55.965420 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.966174 21698 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.966460 23051 consensus_queue.cc:237] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.968686 22831 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:55.968755 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.969391 21701 raft_consensus.cc:2371] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2. Re-sending same reply.
I20250114 08:58:55.970248 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.953583 22241 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:55.970916 21701 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:55.971727 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.973444 23047 consensus_queue.cc:237] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.975745 23083 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.976198 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:55.922616 23071 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:55.977751 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:55.994025 21703 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:55.995079 21697 raft_consensus.cc:2388] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:55.997902 21548 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:55.964699 23097 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.003566 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:56.004222 21696 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:56.005151 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:56.005717 21696 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:56.005893 22989 consensus_meta.cc:287] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.846s user 0.007s sys 0.004s
I20250114 08:58:55.998862 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.985080 21669 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:56.009124 21521 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:56.002287 21622 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:56.009199 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:56.000954 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.013041 21555 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:56.014011 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:55.956876 22673 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:56.018183 21696 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:56.027981 21696 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
W20250114 08:58:56.028353 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 68057.2us
I20250114 08:58:56.023311 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.029556 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.030711 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.031097 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.017321 21698 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:56.018954 21669 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:56.032660 21625 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:58:55.976826 21702 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:56.035219 21552 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:55.979290 23054 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:56.095969 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 66921.8us
I20250114 08:58:55.979900 23121 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:56.127315 23053 consensus_queue.cc:1046] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:56.128854 23053 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:56.129258 22864 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.130273 23053 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:56.131794 23053 consensus_queue.cc:1035] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:55.977962 22974 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.099802 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.027097 22881 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:56.101615 22994 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:56.102407 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.103068 23074 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:58:56.027117 22673 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.109050 22943 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:56.023245 23108 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 08:58:56.112653 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 93770.7us
I20250114 08:58:56.113135 22861 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:56.024194 22872 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:56.167035 22943 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.181353 23083 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.182440 22943 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.024767 23079 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20250114 08:58:56.114831 22831 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.144656 22881 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:56.115430 22891 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.121410 23090 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:55.976856 22848 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:55.979508 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:55.921499 23095 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.146839 21521 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:56.202260 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 55532.1us
I20250114 08:58:56.028787 23070 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:56.151540 21552 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:56.205195 21703 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:56.205807 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.143568 21548 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:56.144084 21550 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:56.207078 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.164626 21669 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:56.208055 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 43529.6us
I20250114 08:58:56.208096 21594 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:56.172562 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:56.152258 22843 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250114 08:58:56.177271 21555 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:56.166117 22861 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.178521 22896 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.155193 23101 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250114 08:58:56.155655 22832 raft_consensus.cc:3049] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:56.163570 22673 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.144258 23052 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250114 08:58:56.167228 23108 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.207146 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.163265 21596 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:56.213696 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.203491 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:56.165583 22974 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250114 08:58:56.155108 22994 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.189563 22963 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.190034 23097 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.215656 21669 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:56.216683 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:56.571575 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:56.572918 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 356315us
I20250114 08:58:56.313951 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.397672 21696 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:56.572980 21550 raft_consensus.cc:2371] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2. Re-sending same reply.
I20250114 08:58:56.417181 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:56.529266 23126 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20250114 08:58:56.580915 21700 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:56.423039 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1128) took 1082 ms. Trace:
I20250114 08:58:56.587184 21552 raft_consensus.cc:2363] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4.
W20250114 08:58:56.587335 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 383915us
I20250114 08:58:56.191942 22881 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:56.592677 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.609230 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.610177 21555 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:56.619416 21548 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:56.584704 21551 rpcz_store.cc:276] 0114 08:58:55.340895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.549535 (+208640us) service_pool.cc:225] Handling call
0114 08:58:55.549884 (+ 349us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:55.842090 (+292206us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:56.422653 (+580563us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b540007c6d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:56.422916 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":610697984}
W20250114 08:58:56.569288 23136 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
W20250114 08:58:56.529150 23127 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
I20250114 08:58:56.192135 22864 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:56.481721 23132 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20250114 08:58:56.632918 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.635334 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.214327 22885 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:56.340673 23054 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:58:56.349886 22891 consensus_queue.cc:1046] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.299s
I20250114 08:58:56.638684 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.585842 22896 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.585676 22861 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.333652 23121 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:56.334013 23134 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:56.585566 23108 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:56.574640 22832 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:56.192847 22831 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:56.647795 23140 log.cc:926] Time spent T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.237s user 0.000s sys 0.000s
I20250114 08:58:56.585386 22943 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:56.697093 22943 raft_consensus.cc:695] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:56.698107 22943 consensus_queue.cc:237] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:56.338367 22848 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.586400 22989 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:58:56.708328 21549 consensus_meta.cc:287] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.748s user 0.009s sys 0.000s
I20250114 08:58:56.609675 23079 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:56.585460 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.719638 23079 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:56.196658 23090 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.720717 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.722008 23090 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:56.655308 23139 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20250114 08:58:56.333876 23133 consensus_queue.cc:1046] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.322s
I20250114 08:58:56.390637 22872 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:56.720825 23079 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.726778 22872 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.727216 22872 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:58:56.626420 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 145195us
W20250114 08:58:56.673796 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 458161us
I20250114 08:58:56.674026 21550 raft_consensus.cc:2388] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:56.675346 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.680353 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.653467 23083 raft_consensus.cc:2758] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250114 08:58:56.683440 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.201337 23071 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.677079 22885 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:56.676566 23054 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:56.710608 23100 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.711169 22831 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.711094 22989 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:56.712044 23078 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:56.712375 23142 log.cc:926] Time spent T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20250114 08:58:56.659830 23051 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20250114 08:58:56.649319 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.724059 21549 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:56.734766 21548 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:56.803972 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.805846 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.748631 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:56.807130 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 58602.7us
I20250114 08:58:56.756268 23095 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.722049 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.812376 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.813251 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.814761 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.768659 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.815510 21550 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:56.816711 21551 raft_consensus.cc:2405] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:56.776178 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.685946 23121 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:56.778324 23144 log.cc:926] Time spent T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250114 08:58:56.784859 23148 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:56.785223 23147 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:56.757737 23073 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20250114 08:58:56.822530 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.814857 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.807130 22994 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.693814 23134 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.766481 21555 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:56.829757 21551 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:56.821027 23147 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:56.833703 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.817605 23083 raft_consensus.cc:3049] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:56.818476 21546 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.831786 23147 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.842921 22994 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:56.843470 22994 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.831450 22271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48992 (request call id 158) took 1650 ms. Trace:
I20250114 08:58:56.831372 23121 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.845441 22994 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.844744 22271 rpcz_store.cc:276] 0114 08:58:55.180822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.226961 (+ 46139us) service_pool.cc:225] Handling call
0114 08:58:55.949580 (+722619us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:58:56.830440 (+880860us) spinlock_profiling.cc:243] Waited 7.82 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656dac 00007fa63c656d16 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:58:56.830611 (+ 171us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:56.830835 (+ 224us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:56.830881 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:56.831070 (+ 189us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:58:56.831124 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:56.831205 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:56.831352 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28479616}
I20250114 08:58:56.847126 23121 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:56.847642 23121 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:56.805976 23078 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:56.850066 23078 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.850378 23121 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.831372 23134 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.852092 21548 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:56.853405 21548 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:56.749167 22832 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.872924 21553 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250114 08:58:56.878942 21552 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:56.828228 23095 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:56.802165 23100 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.836858 23097 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.837512 23083 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:56.882874 21555 raft_consensus.cc:1235] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:56.895695 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.874369 22963 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.915355 22271 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:58:56.917003 22989 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.823161 23148 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.923391 23148 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.926389 23100 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.836441 23054 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.928077 22881 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.902400 23097 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:56.940775 23061 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:56.941716 21550 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:56.942643 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.940798 23097 raft_consensus.cc:695] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:56.945086 21546 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:56.945806 23097 consensus_queue.cc:237] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.947026 21550 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
W20250114 08:58:56.947136 23145 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20250114 08:58:56.945798 21548 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:56.949111 23108 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.950210 21701 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:56.951766 23108 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.883702 22832 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:56.981271 21549 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:56.907400 22861 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.873873 22885 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.889796 23095 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.902424 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.894085 23121 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:56.984890 22861 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.984750 23121 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:56.985545 23095 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.986810 23121 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:56.910236 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:56.987673 21555 raft_consensus.cc:2363] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:58:56.987601 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:56.988428 21696 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:56.931041 23071 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:56.933403 23120 raft_consensus.cc:3049] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:58:56.989529 23120 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:56.990535 21696 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:58:56.990454 23120 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:56.991961 23120 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:56.901715 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:56.992311 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:56.992775 21551 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:58:56.992944 23095 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:56.936692 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 310240us
I20250114 08:58:56.993791 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:56.993563 23095 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:56.939894 21547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1113) took 1923 ms (client timeout 1731 ms). Trace:
I20250114 08:58:56.992388 21704 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:56.939141 23147 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:56.995162 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:56.995273 21547 rpcz_store.cc:269] 0114 08:58:55.016408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.026044 (+ 9636us) service_pool.cc:225] Handling call
0114 08:58:56.939659 (+1913615us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54003a16d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:56.939783 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1405314944}
I20250114 08:58:56.995833 21696 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:56.997098 23147 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.997153 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:56.998193 23147 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:56.998553 23147 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:56.998734 23095 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:56.998895 23147 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.001770 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.002568 21553 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.004164 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.005158 21551 raft_consensus.cc:2388] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.017117 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.017824 21696 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:56.906714 22872 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.020684 22239 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:57.025908 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.027879 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.028923 21550 raft_consensus.cc:2371] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4. Re-sending same reply.
I20250114 08:58:56.924093 23148 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250114 08:58:56.980865 23108 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.018882 23150 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:56.988276 22241 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:57.045903 21521 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:57.043686 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.047348 21703 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:57.058948 21521 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:57.030164 23073 consensus_queue.cc:1046] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250114 08:58:57.070763 21521 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:57.073961 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 919617us
I20250114 08:58:57.077363 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.077831 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.078594 21696 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:57.051329 22837 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:57.085074 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:57.085526 21668 leader_election.cc:336] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.656s (SENT)
I20250114 08:58:57.050865 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.085994 21668 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:57.089766 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.096205 21698 raft_consensus.cc:2388] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.090785 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.092505 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1086) took 1200 ms. Trace:
I20250114 08:58:57.096993 21549 rpcz_store.cc:276] 0114 08:58:55.891901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:56.821875 (+929974us) service_pool.cc:225] Handling call
0114 08:58:56.822125 (+ 250us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:57.020174 (+198049us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.092186 (+ 72012us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b540007bcd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:57.092405 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365432704}
I20250114 08:58:57.099119 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.100339 21698 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:57.107815 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.108636 21703 raft_consensus.cc:2388] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.125471 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1082) took 1234 ms. Trace:
I20250114 08:58:57.125931 21552 rpcz_store.cc:276] 0114 08:58:55.890971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:56.592250 (+701279us) service_pool.cc:225] Handling call
0114 08:58:57.096576 (+504326us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540007d5d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:57.125392 (+ 28816us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":298853120}
I20250114 08:58:57.128278 23155 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:57.128779 23155 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:57.129134 23155 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:57.130529 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 142518us
I20250114 08:58:57.085107 22837 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:57.132876 22837 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:57.132623 23153 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20250114 08:58:57.133850 23153 raft_consensus.cc:3049] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:57.134192 23153 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:57.134981 23153 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.136845 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.137831 21702 raft_consensus.cc:2388] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.056123 23150 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.138495 22896 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.141147 22837 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.141405 23147 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.143154 22837 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:57.143357 22872 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.084861 23078 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:57.145848 23078 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:57.145956 23165 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.146328 23078 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.146442 23165 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:57.149500 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 64004.2us
W20250114 08:58:57.150185 22673 consensus_meta.cc:287] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.532s user 0.005s sys 0.005s
I20250114 08:58:57.150719 23164 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.150684 22673 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.151279 23164 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.151736 21700 raft_consensus.cc:1235] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:57.056747 23108 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.107506 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.154498 21697 raft_consensus.cc:2371] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2. Re-sending same reply.
I20250114 08:58:57.060493 23154 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:58:57.154520 21704 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:57.154909 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.155339 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1323) took 1173 ms. Trace:
I20250114 08:58:57.159399 21701 rpcz_store.cc:276] 0114 08:58:55.920896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.988290 (+ 67394us) service_pool.cc:225] Handling call
0114 08:58:56.022743 (+ 34453us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:56.950562 (+927819us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.093762 (+143200us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b5400129154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:57.093943 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2132557696}
I20250114 08:58:57.160159 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.154305 21695 raft_consensus.cc:2371] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3. Re-sending same reply.
W20250114 08:58:57.161226 21695 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1052) took 1257 ms (client timeout 1510 ms). Trace:
I20250114 08:58:57.155802 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.161782 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.162147 21621 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:58:57.067373 22989 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.162593 21626 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:57.163057 22989 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.032403 23083 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:57.161629 21695 rpcz_store.cc:269] 0114 08:58:55.903729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.903884 (+ 155us) service_pool.cc:225] Handling call
0114 08:58:57.045316 (+1141432us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800297398. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656d0e 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:58:57.161135 (+115819us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26306304}
I20250114 08:58:57.165306 22881 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:57.150157 21668 leader_election.cc:336] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.662s (SENT)
I20250114 08:58:57.166201 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.166570 21668 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:57.166814 21546 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:57.154328 22272 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:58:57.168154 22240 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:57.169124 22891 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:57.170075 23121 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:57.170606 23121 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.168346 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.158605 23154 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:57.172398 21550 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:57.172845 23154 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:58:57.173890 22239 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:57.136282 23153 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.060292 22994 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20250114 08:58:57.107638 23090 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:57.185740 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1321) took 1817 ms. Trace:
I20250114 08:58:57.188866 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.180280 21698 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:57.190925 21699 rpcz_store.cc:276] 0114 08:58:55.368290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.368548 (+ 258us) service_pool.cc:225] Handling call
0114 08:58:55.943792 (+575244us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b5800297398. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:58:56.405832 (+462040us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b5800297398. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656cbc 00007fa63c65687b 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:58:56.406050 (+ 218us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:56.715278 (+309228us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:56.715324 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:56.730388 (+ 15064us) log.cc:844] Serialized 26 byte log entry
0114 08:58:56.985591 (+255203us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:56.985628 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:56.985689 (+ 61us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:57.185518 (+199829us) raft_consensus.cc:1680] finished
0114 08:58:57.185537 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:57.185680 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":378369,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"spinlock_wait_cycles":19148288,"thread_start_us":521147,"threads_started":2,"wal-append.queue_time_us":303546}
I20250114 08:58:57.143345 23147 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.194157 21550 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:57.195919 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.165306 22989 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:57.196852 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:57.201014 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 197598us
I20250114 08:58:57.203051 21703 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.185153 23051 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.185611 23061 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:57.209968 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1097) took 1235 ms. Trace:
I20250114 08:58:57.210269 21548 rpcz_store.cc:276] 0114 08:58:55.974480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:56.854102 (+879622us) service_pool.cc:225] Handling call
0114 08:58:56.945636 (+ 91534us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:56.948951 (+ 3315us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.209827 (+260876us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b5400194950. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:57.209922 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":209800960}
I20250114 08:58:57.204507 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.211216 21621 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.204989 21622 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 08:58:57.189623 23165 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.216954 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.214435 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.217300 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.190872 23090 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:57.181366 23073 consensus_queue.cc:1046] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250114 08:58:57.198346 22891 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:57.217914 21699 raft_consensus.cc:2388] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.214550 21698 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250114 08:58:57.224895 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.215507 23051 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.226928 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.229357 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.202773 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:57.233009 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.224895 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.232411 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.350821 22272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 122) took 1484 ms. Trace:
I20250114 08:58:57.364992 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.245946 21621 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:57.428031 21552 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 08:58:57.496034 21697 raft_consensus.cc:2388] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.234097 21626 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:57.213768 23167 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:58:57.234117 21668 leader_election.cc:336] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.769s (SENT)
I20250114 08:58:57.355278 21451 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 5 to 6, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 6 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.553942 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.554428 21668 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:57.555712 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 321603us
I20250114 08:58:57.379715 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:57.557158 21668 leader_election.cc:336] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.815s (SENT)
I20250114 08:58:57.557286 21701 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:57.557636 21668 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:57.558187 21696 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.261020 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.496920 21627 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:57.356320 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1121) took 2127 ms. Trace:
I20250114 08:58:57.334040 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.560221 21698 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.560495 21622 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:57.560384 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:57.439639 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 67988.3us
I20250114 08:58:57.498634 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.437945 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.443460 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.353323 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.369246 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1106) took 1460 ms. Trace:
I20250114 08:58:57.581760 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.583158 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.555440 21451 catalog_manager.cc:5526] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.583981 21552 pending_rounds.cc:77] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 1
I20250114 08:58:57.584365 21552 pending_rounds.cc:99] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:58:57.584800 21552 raft_consensus.cc:2883] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:58:57.360551 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.594571 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.584412 21451 catalog_manager.cc:5526] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.366396 21629 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.601315 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.601563 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:57.274158 22943 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.011s sys 0.024s
I20250114 08:58:57.231241 22896 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.559167 21624 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.554180 23054 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:57.376092 23143 log.cc:926] Time spent T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.606s user 0.001s sys 0.001s
I20250114 08:58:57.606879 21630 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:57.369526 23079 raft_consensus.cc:3049] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:57.562326 21704 raft_consensus.cc:2388] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:57.561930 22272 rpcz_store.cc:276] 0114 08:58:55.866136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.870136 (+ 4000us) service_pool.cc:225] Handling call
0114 08:58:55.870457 (+ 321us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:57.167449 (+1296992us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.350543 (+183094us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400670554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:57.350750 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2951645056}
I20250114 08:58:57.586165 21625 rpcz_store.cc:276] 0114 08:58:55.909136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.909700 (+ 564us) service_pool.cc:225] Handling call
0114 08:58:55.923397 (+ 13697us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:56.097433 (+174036us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.368889 (+1271456us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540039e754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:57.369143 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250481024}
I20250114 08:58:57.362309 23150 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.376900 23149 raft_consensus.cc:3049] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:57.631525 23149 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:57.632376 23149 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.632895 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:57.633607 23149 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.633244 23078 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.382793 23083 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.609166 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.363111 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.493472 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.499336 23090 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:57.374738 23146 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.645s user 0.004s sys 0.000s
I20250114 08:58:57.559033 21554 rpcz_store.cc:276] 0114 08:58:55.228637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.228801 (+ 164us) service_pool.cc:225] Handling call
0114 08:58:55.247918 (+ 19117us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:55.336051 (+ 88133us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 08:58:55.750332 (+414281us) log.cc:844] Serialized 14 byte log entry
0114 08:58:56.097842 (+347510us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:56.097923 (+ 81us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:56.097980 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.082258 (+984278us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b18002128c0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c661ec6 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9
0114 08:58:57.355969 (+273711us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b540024f9d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:57.356048 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:57.356219 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58188,"prepare.queue_time_us":177921,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":84612,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"spinlock_wait_cycles":239458560,"thread_start_us":623885,"threads_started":2,"wal-append.queue_time_us":443090,"wal-append.run_cpu_time_us":1203,"wal-append.run_wall_time_us":1129175}
I20250114 08:58:57.208891 22673 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.388338 23134 raft_consensus.cc:3049] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:57.656095 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.560416 21550 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:58:57.619882 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.657475 21594 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:57.607260 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:57.618249 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 61118.1us
I20250114 08:58:57.657708 21701 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
W20250114 08:58:57.658394 21668 leader_election.cc:336] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.819s (SENT)
I20250114 08:58:57.637365 23165 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:57.554863 23167 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:57.604816 21623 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:57.623545 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.276607 23157 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.601698 22925 raft_consensus.cc:3049] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:57.628438 21628 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:57.629083 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.608569 23047 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.630154 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.725689 21624 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:57.751586 21699 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:57.617275 23061 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:57.606328 23133 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:57.638924 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.754113 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:58:57.620920 22943 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s user 0.017s sys 0.027s
I20250114 08:58:57.602031 22885 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.639969 21596 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:57.607937 22831 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.432899 23121 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.643431 23164 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:57.620884 23141 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.622064 23152 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.399811 23155 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.453043 22994 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.174s
W20250114 08:58:57.607165 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 413670us
I20250114 08:58:57.645483 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.652503 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.619218 21451 catalog_manager.cc:5526] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.658816 21668 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:57.814939 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 156565us
I20250114 08:58:57.401214 23108 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:57.626292 23079 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:57.644023 23052 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:57.655279 23134 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:57.663175 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:57.663534 22832 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.682s user 0.004s sys 0.006s
I20250114 08:58:57.669168 22843 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:57.670372 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.818593 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.764482 23147 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.673633 23158 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.620756 23073 consensus_queue.cc:1046] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20250114 08:58:57.629074 22768 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:57.709853 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.711674 22266 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:57.651521 21547 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:57.674966 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.738478 21697 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:57.629771 23054 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.752854 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.752759 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.751082 23071 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:57.756812 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:57.834885 21699 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:57.834969 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:57.761216 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.835793 21697 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:57.761343 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:57.763357 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 123468us
I20250114 08:58:57.837458 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.838078 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:57.750953 22925 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:57.739931 21695 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:58:57.773825 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:57.775087 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.810729 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.837209 21624 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:58:57.754007 23061 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:57.818641 22994 raft_consensus.cc:3049] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:57.738018 23175 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:57.815024 21451 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 2 to 3, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.763696 23167 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:57.776666 23154 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.600s user 0.008s sys 0.005s
I20250114 08:58:57.752562 23157 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.826092 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.829176 21521 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:57.849655 21625 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:57.833272 21628 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.850469 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.833681 21623 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:57.851055 21554 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.754882 23047 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.852104 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:57.832973 21550 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:57.819537 23155 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:57.821242 23151 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:57.821233 22843 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:57.821895 22891 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:57.771693 22831 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.772644 22885 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:57.823244 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 58985us
I20250114 08:58:57.847227 21596 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:57.898883 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 51754.8us
I20250114 08:58:57.899596 21596 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:57.848554 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.900498 21626 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:57.848675 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.854880 21627 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:57.854313 21697 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:57.841550 23073 raft_consensus.cc:3049] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:57.856338 21451 catalog_manager.cc:5526] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.818181 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:57.844696 23061 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:57.882539 21628 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:57.867831 22881 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:57.891474 21621 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.842140 23071 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.826407 23147 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.895174 21622 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:57.870922 23078 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:57.869223 23051 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:57.851943 23052 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:58:57.953650 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 54141.2us
I20250114 08:58:57.833200 22768 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.919921 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.818693 23079 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:57.819629 23134 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
W20250114 08:58:57.992933 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 272459us
I20250114 08:58:57.914983 23073 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:57.922958 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:57.994905 21626 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:57.891985 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.899958 21623 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:57.996114 21629 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.913336 21451 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 5, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 5 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:57.996982 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:57.917250 23101 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:58:57.845731 22994 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:57.890612 22843 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:57.997658 21623 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:57.849201 23154 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:57.923422 22832 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:57.964852 21621 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
W20250114 08:58:57.998260 23174 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.278s user 0.001s sys 0.000s
I20250114 08:58:58.012967 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.020949 21626 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:57.950273 23164 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:58:58.000965 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 283586us
I20250114 08:58:57.954079 23052 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:57.949975 22881 raft_consensus.cc:695] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:57.778431 23141 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.042476 21629 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:58.035652 21623 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:58.038157 21521 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:58:57.992555 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 145672us
I20250114 08:58:58.040499 22274 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:58:58.011008 23134 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:58.049441 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 94951.3us
I20250114 08:58:57.950898 22241 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:58.026816 21701 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:58.053104 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.053331 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.053848 21626 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:58.048950 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 233398us
W20250114 08:58:58.054863 21668 leader_election.cc:336] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.002s (SENT)
I20250114 08:58:58.054714 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.052148 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:57.828524 23158 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.058190 21629 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:57.848630 23175 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.024513 23079 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.046169 21623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:57318
I20250114 08:58:57.817945 23152 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.029520 23164 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:58.085326 22240 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:58.085610 23164 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.067451 23181 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:58.089540 21704 raft_consensus.cc:1270] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20250114 08:58:58.089954 21546 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.923s user 0.008s sys 0.003s
I20250114 08:58:58.090528 21546 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:58.091800 21700 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:58:58.092579 21625 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:58:58.092950 23152 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.067427 21623 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:58:58.097246 21623 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:58.075538 23157 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.048605 22832 raft_consensus.cc:695] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:58.039469 22831 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:58.110878 23173 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.289s user 0.001s sys 0.001s
I20250114 08:58:58.111397 22989 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.112254 23185 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.112783 23185 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.112957 23164 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.115020 23164 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.114948 23185 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.101861 23181 raft_consensus.cc:695] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:58.080869 23158 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.116674 23157 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.116576 23181 consensus_queue.cc:237] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:58.117775 23178 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250114 08:58:58.082963 23175 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.126974 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1078) took 1020 ms. Trace:
I20250114 08:58:58.127391 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.084787 22266 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:58:58.084506 23090 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.082984 23154 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.085429 23147 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:58.087939 22881 consensus_queue.cc:237] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:58.093058 23172 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.373s user 0.001s sys 0.001s
I20250114 08:58:58.049438 23073 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.089970 21704 pending_rounds.cc:77] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:58:58.153092 21704 pending_rounds.cc:99] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:58:58.153399 21704 raft_consensus.cc:2883] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250114 08:58:58.152534 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 100473us
I20250114 08:58:57.915561 22925 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.043876 23052 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:58.066236 21668 leader_election.cc:336] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.512s (SENT)
I20250114 08:58:58.100955 21629 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:58:58.161346 21668 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:58.118883 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1077) took 1013 ms. Trace:
I20250114 08:58:58.121294 21554 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:58.062244 22994 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.123683 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1082) took 1015 ms. Trace:
I20250114 08:58:58.125499 22832 consensus_queue.cc:237] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.129395 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.129459 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.164297 21701 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:58:58.164399 21549 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:58:58.141415 21626 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:58.067179 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:58.118548 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 197074us
I20250114 08:58:58.127526 21696 rpcz_store.cc:276] 0114 08:58:57.106193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.606277 (+500084us) service_pool.cc:225] Handling call
0114 08:58:57.606642 (+ 365us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:57.668884 (+ 62242us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:57.668914 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:57.669372 (+ 458us) log.cc:844] Serialized 26 byte log entry
0114 08:58:57.826178 (+156806us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:57.843882 (+ 17704us) log.cc:844] Serialized 14 byte log entry
0114 08:58:57.844145 (+ 263us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.844232 (+ 87us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.126713 (+282481us) raft_consensus.cc:1680] finished
0114 08:58:58.126738 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.126891 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88055,"prepare.queue_time_us":17477,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":51795,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":42716,"spinlock_wait_cycles":6153344,"thread_start_us":156453,"threads_started":1,"wal-append.queue_time_us":353925}
I20250114 08:58:58.166393 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:58.144497 21623 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.152148 21627 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250114 08:58:58.130509 23078 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:58.155964 21596 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.158680 23073 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.155269 22885 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.159715 23090 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.159919 22925 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.160578 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.119732 23054 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.161996 21703 rpcz_store.cc:276] 0114 08:58:57.105906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.594659 (+488753us) service_pool.cc:225] Handling call
0114 08:58:57.624850 (+ 30191us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:57.648833 (+ 23983us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:57.648879 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:57.649417 (+ 538us) log.cc:844] Serialized 26 byte log entry
0114 08:58:57.912277 (+262860us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:57.948056 (+ 35779us) log.cc:844] Serialized 14 byte log entry
0114 08:58:57.955436 (+ 7380us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.955549 (+ 113us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.118682 (+163133us) raft_consensus.cc:1680] finished
0114 08:58:58.118699 (+ 17us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.118825 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47861,"prepare.queue_time_us":81602,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":26774,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":28431,"thread_start_us":191887,"threads_started":1,"wal-append.queue_time_us":400861}
W20250114 08:58:58.162194 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 116855us
W20250114 08:58:58.166509 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 100427us
I20250114 08:58:58.169234 21626 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:58:58.166762 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:58.178129 21668 leader_election.cc:336] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.533s (SENT)
W20250114 08:58:58.179013 21668 leader_election.cc:336] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.051s (SENT)
I20250114 08:58:58.179277 21668 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:58.169216 21623 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:58.170045 23153 consensus_meta.cc:287] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.979s user 0.011s sys 0.000s
I20250114 08:58:58.146981 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:58.167052 23177 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20250114 08:58:58.173266 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.185165 21701 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:58.175130 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.148046 22271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48992 (request call id 165) took 1007 ms. Trace:
W20250114 08:58:58.156558 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 92179.6us
I20250114 08:58:58.167269 23047 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.175791 21629 raft_consensus.cc:1235] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.186287 21548 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:58.187412 21627 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:58.168561 23147 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:58.168838 23078 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:58.203768 21549 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:58:58.198046 21596 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.200390 21626 raft_consensus.cc:1235] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:58.179874 23101 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:58.173035 22994 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:58.172883 22837 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:58:58.205511 21703 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:58.207471 21701 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:58:58.209782 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.188818 23071 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.221346 21627 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:58.223913 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.210976 21701 raft_consensus.cc:1235] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.298110 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1193) took 1076 ms. Trace:
I20250114 08:58:58.298269 21552 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:58:58.262658 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.210240 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.298560 21627 rpcz_store.cc:276] 0114 08:58:57.222182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.209311 (+987129us) service_pool.cc:225] Handling call
0114 08:58:58.221005 (+ 11694us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.297743 (+ 76738us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.298051 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:58:58.255919 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57919.4us
I20250114 08:58:58.222836 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1190) took 1001 ms. Trace:
I20250114 08:58:58.221113 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1113) took 1007 ms. Trace:
I20250114 08:58:58.211613 21629 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:58.300338 21623 rpcz_store.cc:276] 0114 08:58:57.221215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.201732 (+980517us) service_pool.cc:225] Handling call
0114 08:58:58.222371 (+ 20639us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.222543 (+ 172us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:58.222567 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:58.222606 (+ 39us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:58.222635 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.222686 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.222772 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:58.302484 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1196) took 1079 ms. Trace:
I20250114 08:58:58.302974 21629 rpcz_store.cc:276] 0114 08:58:57.223096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.211195 (+988099us) service_pool.cc:225] Handling call
0114 08:58:58.211439 (+ 244us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.302150 (+ 90711us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.302411 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:58:58.209206 22837 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:58.312640 22837 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:58.313287 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:58.223116 23195 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20250114 08:58:58.314946 21701 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:58.214228 23164 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:58.300520 21621 rpcz_store.cc:276] 0114 08:58:57.213353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.582910 (+369557us) service_pool.cc:225] Handling call
0114 08:58:58.220946 (+638036us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54000c8354. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:58.221045 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707566848}
I20250114 08:58:58.317564 21621 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:58:58.311969 23151 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.174443 23167 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.319510 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1195) took 1096 ms. Trace:
I20250114 08:58:58.197824 21546 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:58.325609 21626 rpcz_store.cc:276] 0114 08:58:57.222782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.208592 (+985810us) service_pool.cc:225] Handling call
0114 08:58:58.319051 (+110459us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.319218 (+ 167us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:58.319243 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:58.319283 (+ 40us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:58.319307 (+ 24us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.319358 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.319455 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:58:58.261521 23083 consensus_meta.cc:287] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.625s user 0.011s sys 0.000s
I20250114 08:58:58.206615 23078 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.330451 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:58.334770 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 125071us
I20250114 08:58:58.206975 23193 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:58.339825 21628 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:58.225389 22768 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.224977 23141 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.346076 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.343930 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1201) took 1087 ms. Trace:
I20250114 08:58:58.348126 21623 rpcz_store.cc:276] 0114 08:58:57.257059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.302384 (+1045325us) service_pool.cc:225] Handling call
0114 08:58:58.333680 (+ 31296us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.343492 (+ 9812us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:58.343535 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:58:58.343590 (+ 55us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:58.343634 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.343713 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.343860 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9439,"raft.queue_time_us":9605}
I20250114 08:58:58.349710 21596 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.350157 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.352614 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.316085 23164 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.359277 21596 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.366207 21550 raft_consensus.cc:1235] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:58.368610 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.373968 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.374703 21550 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:58.375870 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.341920 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.382676 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.382676 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.384629 21550 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:58.389544 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.390983 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.219617 22843 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.259501 23061 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.392385 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.338766 23074 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:58:58.215791 22241 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.396538 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.345990 23054 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:58.400352 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.404606 21554 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:58.356981 23051 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 08:58:58.405005 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 45871.3us
I20250114 08:58:58.410001 21630 raft_consensus.cc:2388] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:58.410473 21596 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.411749 21596 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.409399 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.412817 21626 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:58:58.402971 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.406697 21594 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.407604 21555 raft_consensus.cc:2363] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:58.376808 23083 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.402935 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.415478 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.369849 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.417356 21552 raft_consensus.cc:2405] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:58.420887 23175 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:58.425974 21621 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250114 08:58:58.426653 23196 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
W20250114 08:58:58.417793 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 202085us
I20250114 08:58:58.189649 22271 rpcz_store.cc:276] 0114 08:58:57.140312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.140501 (+ 189us) service_pool.cc:225] Handling call
0114 08:58:57.140864 (+ 363us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:57.141243 (+ 379us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:57.141297 (+ 54us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:57.141793 (+ 496us) log.cc:844] Serialized 26 byte log entry
0114 08:58:57.619529 (+477736us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:57.625664 (+ 6135us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:57.630808 (+ 5144us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.118628 (+487820us) raft_consensus.cc:1680] finished
0114 08:58:58.128788 (+ 10160us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.147957 (+ 19169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1147,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":459419,"threads_started":1,"wal-append.queue_time_us":611123}
I20250114 08:58:58.431337 23167 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.437567 23061 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.409389 23101 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:58.440994 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1086) took 1210 ms. Trace:
I20250114 08:58:58.441403 21699 rpcz_store.cc:276] 0114 08:58:57.230174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.842369 (+612195us) service_pool.cc:225] Handling call
0114 08:58:57.857577 (+ 15208us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:57.858000 (+ 423us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:57.858041 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:58.053734 (+195693us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b2c00042708. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63dfae741 00007fa63dfbdd72 00007fa63f609141 00007fa63f5cd58a 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782
0114 08:58:58.092655 (+ 38921us) log.cc:844] Serialized 26 byte log entry
0114 08:58:58.147637 (+ 54982us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:58.172968 (+ 25331us) log.cc:844] Serialized 14 byte log entry
0114 08:58:58.440516 (+267548us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b18003ccd60. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c60c03a 00007fa645ac8be9 00007fa63c4cd930 00007fa63c691a14 00007fa63c691305 00007fa63f5c53d0 00007fa63f5d7666 00007fa63f5e81d7 00007fa63f5e8158
0114 08:58:58.440693 (+ 177us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.440760 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.440800 (+ 40us) raft_consensus.cc:1680] finished
0114 08:58:58.440812 (+ 12us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.440935 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122654,"prepare.queue_time_us":25129,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":4910,"raft.queue_time_us":11577,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":26067,"spinlock_wait_cycles":21020160,"thread_start_us":54644,"threads_started":1,"wal-append.queue_time_us":80699}
I20250114 08:58:58.444144 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1083) took 1335 ms. Trace:
I20250114 08:58:58.165511 21702 rpcz_store.cc:276] 0114 08:58:57.108562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.637237 (+528675us) service_pool.cc:225] Handling call
0114 08:58:57.661133 (+ 23896us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:57.661603 (+ 470us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:57.661647 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:57.662290 (+ 643us) log.cc:844] Serialized 26 byte log entry
0114 08:58:57.844385 (+182095us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:58:58.067880 (+223495us) log.cc:844] Serialized 14 byte log entry
0114 08:58:58.090612 (+ 22732us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.090695 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.123448 (+ 32753us) raft_consensus.cc:1680] finished
0114 08:58:58.123475 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.123624 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48923,"prepare.queue_time_us":51090,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":19632,"raft.queue_time_us":7497,"raft.run_cpu_time_us":4801,"raft.run_wall_time_us":36492,"thread_start_us":358554,"threads_started":2,"wal-append.queue_time_us":356823}
I20250114 08:58:58.444675 21704 rpcz_store.cc:276] 0114 08:58:57.108870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.638353 (+529483us) service_pool.cc:225] Handling call
0114 08:58:57.638610 (+ 257us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.154157 (+515547us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.443835 (+289678us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54001c0f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:58.444070 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1036147456}
I20250114 08:58:58.446182 22270 raft_consensus.cc:2388] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:58:58.446815 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 267791us
W20250114 08:58:58.447489 21668 leader_election.cc:336] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.118s (SENT)
I20250114 08:58:58.414961 23193 consensus_queue.cc:1046] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250114 08:58:58.447937 21668 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:58.448925 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1120) took 1229 ms. Trace:
I20250114 08:58:58.449267 21622 rpcz_store.cc:276] 0114 08:58:57.219737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.594252 (+374515us) service_pool.cc:225] Handling call
0114 08:58:58.448766 (+854514us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54000c7e54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:58.448870 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":631234432}
I20250114 08:58:58.369669 23047 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250114 08:58:58.418121 23157 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:58.397639 23141 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.456260 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:58.457729 21668 leader_election.cc:336] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.751s (SENT)
I20250114 08:58:58.477828 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.478749 21623 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:58:58.479044 21668 leader_election.cc:336] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.763s (SENT)
I20250114 08:58:58.455042 22240 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:58.482194 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 59020.7us
W20250114 08:58:58.481068 21668 leader_election.cc:336] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.142s (SENT)
I20250114 08:58:58.485076 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:58:58.486240 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 119141us
I20250114 08:58:58.487128 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.487181 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.487846 21629 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 3
I20250114 08:58:58.447247 23052 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.495113 21629 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:58:58.424089 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.500231 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:58.503495 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 91820.6us
I20250114 08:58:58.455626 23051 raft_consensus.cc:2758] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250114 08:58:58.408087 23074 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.475176 23147 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.509932 21624 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:58.464195 23151 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:58.506737 21596 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.481336 22896 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.496209 23193 raft_consensus.cc:3049] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:58.560948 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1076) took 1455 ms. Trace:
I20250114 08:58:58.494942 23154 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:58.495394 22885 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.562064 21623 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:58:58.562143 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.567507 21626 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:58:58.568105 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 161515us
W20250114 08:58:58.562021 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 74961.9us
I20250114 08:58:58.568845 21594 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.569154 21669 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:58:58.569559 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 62928us
I20250114 08:58:58.570328 21596 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:58.504644 23181 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:58:58.571341 21594 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.573731 21668 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:58.626658 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 54493.8us
I20250114 08:58:58.578761 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.499615 23141 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.625288 22861 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.633113 22861 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.633673 23074 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.634167 23141 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.558583 23155 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.634848 23155 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.635931 22861 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.635713 23078 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.636765 23204 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:58.585305 23195 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.637394 23204 raft_consensus.cc:695] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:58.637602 23155 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.637712 23195 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.638482 22272 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250114 08:58:58.638336 23204 consensus_queue.cc:237] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.639423 23078 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.511466 23164 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.639914 23195 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.642108 23175 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:58.625592 22885 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.564779 21698 rpcz_store.cc:276] 0114 08:58:57.105563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.595653 (+490090us) service_pool.cc:225] Handling call
0114 08:58:57.601394 (+ 5741us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:57.627043 (+ 25649us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:57.627091 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:57.645744 (+ 18653us) log.cc:844] Serialized 26 byte log entry
0114 08:58:57.968526 (+322782us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:58:58.021387 (+ 52861us) log.cc:844] Serialized 14 byte log entry
0114 08:58:58.021596 (+ 209us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.021665 (+ 69us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.492688 (+471023us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b1800377ce0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:58:58.494392 (+ 1704us) raft_consensus.cc:1680] finished
0114 08:58:58.513240 (+ 18848us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.560868 (+ 47628us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79260,"prepare.queue_time_us":59006,"prepare.run_wall_time_us":51,"raft.queue_time_us":29174,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":15989,"spinlock_wait_cycles":37911680,"thread_start_us":264365,"threads_started":1,"wal-append.queue_time_us":394897}
I20250114 08:58:58.646203 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:58.558046 22925 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.651425 22885 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.653822 21549 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
W20250114 08:58:58.662626 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 91584.4us
I20250114 08:58:58.674981 21594 leader_election.cc:400] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:58.675417 21594 leader_election.cc:403] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:58:58.569193 23083 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:58.628907 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 50291.8us
I20250114 08:58:58.510006 22891 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
W20250114 08:58:58.668893 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 102278us
I20250114 08:58:58.663600 23164 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.667218 23156 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:58:58.625355 22768 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:58.686427 22768 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:58.686842 22768 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.689085 23052 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.512848 22832 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.149s
I20250114 08:58:58.690913 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.694828 22274 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:58:58.626315 23192 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:58:58.672274 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 103196us
I20250114 08:58:58.657872 23051 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:58:58.678565 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 104932us
I20250114 08:58:58.521037 23185 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.511518 23167 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:58.683445 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.683239 23164 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.684172 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.681730 23156 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:58.685259 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.512280 23157 consensus_queue.cc:1046] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.225s
I20250114 08:58:58.509516 23159 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:58.696444 21669 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.564342 23193 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:58.679034 22896 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:58.757489 22896 raft_consensus.cc:695] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:58.757160 23159 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:58.758421 22896 consensus_queue.cc:237] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.759068 21546 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:58.761572 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.761749 21668 leader_election.cc:400] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:58:58.762224 23183 raft_consensus.cc:3049] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:58.762715 23183 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:58.762957 22270 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:58.763935 23164 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:58.763561 23183 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:58.685946 23197 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.317s user 0.001s sys 0.000s
I20250114 08:58:58.787261 21704 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:58:58.681494 23165 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.788450 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.789067 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.789381 21703 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.788659 21702 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:58:58.570683 23101 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:58.789856 21695 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:58.672655 23158 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20250114 08:58:58.790879 23158 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:58:58.791225 23158 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:58.692044 22843 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:58.745669 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 49368.1us
I20250114 08:58:58.791999 23158 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.792551 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.793274 23158 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:58:58.793859 21669 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.794701 21669 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.795477 21669 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.795847 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1119) took 1577 ms. Trace:
W20250114 08:58:58.748570 23207 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.058s user 0.002s sys 0.000s
I20250114 08:58:58.796329 21624 rpcz_store.cc:276] 0114 08:58:57.219494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.840951 (+621457us) service_pool.cc:225] Handling call
0114 08:58:58.795594 (+954643us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b54000c8854. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:58.795761 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1398002944}
I20250114 08:58:58.797706 23133 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:58.797523 22861 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:58.798244 23133 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:58.799007 22861 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:58.799543 22861 raft_consensus.cc:695] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:58.799216 23133 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.800691 22891 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:58:58.801292 23152 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:58.801335 22891 raft_consensus.cc:695] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:58.801873 23152 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:58.802313 23152 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.802465 22891 consensus_queue.cc:237] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.749475 23147 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:58.752512 23193 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.681932 23083 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.727020 23054 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:58.736418 23051 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:58.740629 23185 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:58.737923 23052 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:58.800451 22861 consensus_queue.cc:237] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.762301 21668 leader_election.cc:403] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250114 08:58:58.744504 23167 raft_consensus.cc:695] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:58.766064 21518 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:58.849272 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:58:58.849990 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 83994.2us
I20250114 08:58:58.767143 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:58.738416 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 75634.2us
I20250114 08:58:58.853231 23193 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:58.853338 23054 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.777012 22837 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.677953 22925 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.777571 22239 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:58.837172 23051 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.698666 23181 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:58:58.784586 23079 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.689s user 0.009s sys 0.002s
I20250114 08:58:58.785526 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.879557 21698 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:58.786381 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:58.880785 21696 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
W20250114 08:58:58.880918 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 119304us
I20250114 08:58:58.882189 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.800755 21627 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.750830 23156 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:58.804854 23149 raft_consensus.cc:3049] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:58.808211 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1089) took 1577 ms. Trace:
I20250114 08:58:58.885593 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.796205 21669 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:58.809654 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1203) took 1214 ms. Trace:
I20250114 08:58:58.902971 21521 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:58.903312 21630 rpcz_store.cc:276] 0114 08:58:57.595590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.670084 (+ 74494us) service_pool.cc:225] Handling call
0114 08:58:58.809458 (+1139374us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b540022d154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:58.809601 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1352327424}
I20250114 08:58:58.810268 21625 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:58:58.806454 23147 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:58.756325 22768 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.860008 21549 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:58:58.861127 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 83643.1us
I20250114 08:58:58.779721 22885 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.862884 21547 raft_consensus.cc:1270] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 08:58:58.863381 22925 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.868598 23154 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:58:58.835374 23175 raft_consensus.cc:695] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:58.870288 22837 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.870836 21551 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:58.838305 23165 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.858392 23195 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:58.835933 23164 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:58:58.880162 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 205365us
I20250114 08:58:58.860560 23183 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.908955 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1197) took 1685 ms. Trace:
I20250114 08:58:58.920811 23155 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:58.902104 21697 rpcz_store.cc:276] 0114 08:58:57.232240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.918040 (+685800us) service_pool.cc:225] Handling call
0114 08:58:57.918388 (+ 348us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:58.094520 (+176132us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:58.094566 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:58.130180 (+ 35614us) log.cc:844] Serialized 26 byte log entry
0114 08:58:58.203619 (+ 73439us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:58:58.405726 (+202107us) log.cc:844] Serialized 14 byte log entry
0114 08:58:58.668762 (+263036us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.768708 (+ 99946us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b18003fc6a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c661ec6 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9
0114 08:58:58.769426 (+ 718us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400346554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:58:58.769475 (+ 49us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:58:58.807929 (+ 38454us) raft_consensus.cc:1680] finished
0114 08:58:58.807958 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:58:58.808136 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":335588,"prepare.queue_time_us":113966,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":263024,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":14574,"spinlock_wait_cycles":407613568,"thread_start_us":133039,"threads_started":2,"wal-append.queue_time_us":187043}
I20250114 08:58:58.913427 21622 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:58.914153 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.915215 21626 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.917332 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.917083 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:58.920223 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.922075 21668 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:58.953830 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.957161 21626 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:58:58.920876 22994 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:58:58.917035 21520 connection.cc:232] RPC call timeout handler was delayed by -1.1470388469999762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
W20250114 08:58:58.963090 21520 connection.cc:232] RPC call timeout handler was delayed by -1.1344388469999558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20250114 08:58:58.879340 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.927106 23071 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20250114 08:58:58.901790 23090 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.928189 21700 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.883385 23108 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:58:58.913681 23147 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:58.903707 23149 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:58.912446 23079 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.937602 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:58.999094 23134 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:58.938306 21548 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:58.938911 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:59.000522 21550 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.000679 23209 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:58.939042 23185 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.940852 21627 rpcz_store.cc:276] 0114 08:58:57.223429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.316541 (+1093112us) service_pool.cc:225] Handling call
0114 08:58:58.800570 (+484029us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.883544 (+ 82974us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.908674 (+ 25130us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54000c8850. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:58.908876 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1112413568}
I20250114 08:58:58.911523 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.917673 23074 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:58.861464 23153 consensus_queue.cc:1046] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20250114 08:58:58.950248 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:58.958168 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.954648 21621 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:59.040511 22272 raft_consensus.cc:2405] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:58:59.040910 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:58.952944 21623 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:58.953343 21628 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:58.961052 21549 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:58.922678 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1117) took 1707 ms. Trace:
I20250114 08:58:59.042263 21549 raft_consensus.cc:604] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.042193 21625 rpcz_store.cc:276] 0114 08:58:57.215602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:58.093277 (+877675us) service_pool.cc:225] Handling call
0114 08:58:58.809598 (+716321us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:58:58.911002 (+101404us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:58.922422 (+ 11420us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b540022d150. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:58:58.922598 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1646552320}
I20250114 08:58:59.043385 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.963390 21551 raft_consensus.cc:1270] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250114 08:58:58.963428 21520 connection.cc:232] RPC call timeout handler was delayed by -1.1200388469999325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
W20250114 08:58:59.045871 21520 connection.cc:232] RPC call timeout handler was delayed by -1.0615388469999516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1628s for a response
W20250114 08:58:59.046244 21520 connection.cc:232] RPC call timeout handler was delayed by -1.0489388469999312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
I20250114 08:58:58.925757 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.964285 21668 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.004381 23079 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:58.969516 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 46991.3us
W20250114 08:58:59.049187 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 84978.9us
I20250114 08:58:58.929893 23195 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:58.988761 23090 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.976608 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.932430 22891 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
W20250114 08:58:58.977666 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 181510us
I20250114 08:58:58.952059 23155 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.072687 23155 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.075313 23181 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:58.925129 22896 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250114 08:58:59.078009 23090 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.924739 22885 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.079984 22885 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.981570 23175 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.082005 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:59.082142 23152 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.933687 22881 consensus_queue.cc:1046] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20250114 08:58:59.083473 22274 raft_consensus.cc:1235] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.978243 21546 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.084149 23152 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:58.933145 23165 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.056808 23074 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.100137 23147 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.100960 23165 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:59.102185 23147 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:58.971268 22994 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:58.996635 23083 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:58.932957 23167 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.036520 21550 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:58:59.048447 23153 raft_consensus.cc:3049] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:59.049846 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:58.931952 23154 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:59.052044 21625 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:58.939508 23158 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
W20250114 08:58:59.122045 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 72190.6us
I20250114 08:58:59.050251 21521 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:59.062116 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:59.062489 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:59.126706 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 76683us
I20250114 08:58:59.069615 21624 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:59.072062 21551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:59.127907 21551 raft_consensus.cc:604] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:58.806886 23159 consensus_queue.cc:1159] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" responder_term: 6 status { last_received { term: 5 index: 6 } last_committed_idx: 6 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20250114 08:58:58.985138 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:59.139887 21552 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:58:59.139927 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.140691 21626 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:59.141522 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.087744 23149 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.088645 23079 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.092455 23220 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.092530 21622 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:59.083624 22881 raft_consensus.cc:3049] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:59.093393 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:58:59.097115 23217 log.cc:926] Time spent T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250114 08:58:59.156423 21621 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:58:59.157757 22994 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:59.098039 23214 log.cc:926] Time spent T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20250114 08:58:58.951097 23164 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.104010 22272 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:59.059919 23195 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.120494 23153 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:59.109242 23157 raft_consensus.cc:3049] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:58:59.064169 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.116150 21623 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
W20250114 08:58:59.119210 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 82098.3us
W20250114 08:58:59.122865 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 69599.9us
I20250114 08:58:59.165498 21630 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:58:59.106048 23074 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:58:59.166006 21520 leader_election.cc:336] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.628s (SENT)
I20250114 08:58:59.059799 23211 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.129699 21521 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.105513 23133 consensus_queue.cc:1046] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20250114 08:58:59.166536 21520 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.084676 22891 raft_consensus.cc:695] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:58:59.129752 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 67384us
I20250114 08:58:59.132938 21668 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.136754 21551 raft_consensus.cc:612] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:58:59.137975 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:59.152930 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.169106 21628 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:58:59.154412 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.060155 23054 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20250114 08:58:59.155663 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.106528 21546 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:58:59.060351 23071 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20250114 08:58:59.141892 22885 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:58:59.128629 22768 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:59.142916 23165 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:59.168547 21669 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:59.155215 22881 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:59.171891 21668 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.167383 23157 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:59.143007 23090 consensus_queue.cc:1046] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250114 08:58:59.153003 23181 consensus_queue.cc:1046] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250114 08:58:59.162458 23052 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:59.198514 22270 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250114 08:58:59.198709 21551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82452b55145444529bf0609cd7767adf"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:59.199843 23108 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:58:59.212834 21629 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:59.211925 21668 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.173480 21521 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.205592 21594 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:59.209408 21551 raft_consensus.cc:604] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.214154 21551 raft_consensus.cc:612] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:58:59.172881 23079 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:59.224840 23079 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.225251 23079 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.225939 23051 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.226644 23149 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.226686 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.227962 23051 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.210325 21669 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:59.194460 23158 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:59.167399 23155 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.214181 23208 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:59.251053 23208 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.252321 23155 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.248073 23219 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:59.253319 23208 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:59.216372 23083 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:58:59.254190 23141 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:59.254585 23141 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.254909 23141 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.207852 22891 consensus_queue.cc:237] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.218554 23181 raft_consensus.cc:3049] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:58:59.257701 23181 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:59.257580 22837 consensus_queue.cc:1046] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:58:59.197049 22843 consensus_queue.cc:1035] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20250114 08:58:59.208408 23157 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.254225 23083 raft_consensus.cc:695] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:59.259150 23152 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:59.259591 23083 consensus_queue.cc:237] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.259720 23152 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.223304 23224 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:59.223899 23108 raft_consensus.cc:695] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:59.192673 23133 consensus_queue.cc:1225] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer 57f9cc1dd3c64647848a33a65b08c893 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20250114 08:58:59.228852 23222 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250114 08:58:59.212870 22896 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.228717 23204 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:58:59.039855 23206 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:58:59.237699 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.231472 22925 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.173300 23074 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20250114 08:58:59.199429 22881 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.242462 23185 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:59.266357 23152 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:59.231638 22885 raft_consensus.cc:3049] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:59.229965 23164 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.202474 23165 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:59.284812 22885 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:59.285637 22885 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.245324 21669 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:59.286965 22885 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.287655 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.288487 22896 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:59.169309 22994 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:59.249681 23221 consensus_queue.cc:1046] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250114 08:58:59.249994 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 76588.3us
I20250114 08:58:59.301988 23157 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:59.302987 23181 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.302248 21521 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.248579 23158 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.304549 23221 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:59.304812 23158 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.304987 23221 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.305177 23220 raft_consensus.cc:3049] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:59.305436 23221 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.305579 23220 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:59.306788 23153 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.307832 23153 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.222751 21700 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
I20250114 08:58:59.308751 21700 raft_consensus.cc:604] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:59.273233 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.172904 23154 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.253381 23219 raft_consensus.cc:695] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:59.278390 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 72) took 3995 ms. Trace:
I20250114 08:58:59.242511 23071 raft_consensus.cc:988] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:58:59.271451 22768 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20250114 08:58:59.279985 23073 consensus_queue.cc:1046] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:58:59.281010 23108 consensus_queue.cc:237] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.278378 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:58:59.285596 23165 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.289127 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:58:59.124604 23156 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.247121 21594 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:59.289961 23224 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:59.287910 23195 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.301189 22896 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:58:59.290868 23152 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.301532 22274 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250114 08:58:59.348941 21700 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:58:59.304354 23181 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:58:59.314353 21551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:58:59.350374 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.350888 21551 raft_consensus.cc:604] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.351064 21696 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:58:59.169432 23211 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.318574 21520 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:59.321528 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 109690us
I20250114 08:58:59.321113 21451 rpcz_store.cc:276] 0114 08:58:55.281540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.281682 (+ 142us) service_pool.cc:225] Handling call
0114 08:58:59.276744 (+3995062us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:58:58.206929 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:58:58.224553 (+ 17624us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:58:58.224590 (+ 37us) write_op.cc:435] Acquired schema lock
0114 08:58:58.224611 (+ 21us) tablet.cc:659] Decoding operations
0114 08:58:58.225515 (+ 904us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:58:58.225572 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0114 08:58:58.225596 (+ 24us) tablet.cc:682] Acquiring locks for 7 operations
0114 08:58:58.226171 (+ 575us) tablet.cc:698] Row locks acquired
0114 08:58:58.226184 (+ 13us) write_op.cc:260] PREPARE: finished
0114 08:58:58.226346 (+ 162us) write_op.cc:270] Start()
0114 08:58:58.226481 (+ 135us) write_op.cc:276] Timestamp: P: 1736845138226293 usec, L: 0
0114 08:58:58.226510 (+ 29us) op_driver.cc:348] REPLICATION: starting
0114 08:58:58.357553 (+131043us) log.cc:844] Serialized 3047 byte log entry
0114 08:58:58.492185 (+134632us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63c67681e 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b
0114 08:58:58.986726 (+494541us) op_driver.cc:464] REPLICATION: finished
0114 08:58:59.202582 (+215856us) write_op.cc:301] APPLY: starting
0114 08:58:59.207901 (+ 5319us) tablet.cc:1364] starting BulkCheckPresence
0114 08:58:59.208115 (+ 214us) tablet.cc:1367] finished BulkCheckPresence
0114 08:58:59.208140 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:58:59.241134 (+ 32994us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:58:59.241330 (+ 196us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0114 08:58:59.241376 (+ 46us) write_op.cc:312] APPLY: finished
0114 08:58:59.241627 (+ 251us) log.cc:844] Serialized 58 byte log entry
0114 08:58:59.241803 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:59.241917 (+ 114us) write_op.cc:454] Released schema lock
0114 08:58:59.241960 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0114 08:58:59.242292 (+ 332us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:58:59.242323 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"spinlock_wait_cycles":19328,"child_traces":[["op",{"apply.queue_time_us":185833,"apply.run_cpu_time_us":3020,"apply.run_wall_time_us":44936,"num_ops":7,"prepare.queue_time_us":136932,"prepare.run_cpu_time_us":10029,"prepare.run_wall_time_us":374179,"raft.queue_time_us":189103,"raft.run_cpu_time_us":18486,"raft.run_wall_time_us":165743,"replication_time_us":760069,"spinlock_wait_cycles":19311872,"thread_start_us":449889,"threads_started":4,"wal-append.queue_time_us":295674}]]}
W20250114 08:58:59.334910 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 119244us
I20250114 08:58:59.338784 21698 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:59.340262 22239 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:59.392134 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 51945.1us
I20250114 08:58:59.346549 23133 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:59.329376 23071 raft_consensus.cc:1076] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Signalling peer d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:58:59.334381 23073 consensus_queue.cc:1225] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:58:59.342406 23183 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:59.393512 22837 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:59.400719 23183 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.395839 23156 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.393237 22270 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:58:59.397397 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:58:59.394323 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.402366 23133 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.402863 21696 raft_consensus.cc:2405] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:58:59.403455 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:58:59.389694 21555 raft_consensus.cc:1235] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
W20250114 08:58:59.341117 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 52069.4us
I20250114 08:58:59.342465 22925 consensus_queue.cc:1046] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20250114 08:58:59.348304 22994 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.415786 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:59.416733 21549 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.401139 23183 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.402805 23133 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.404101 21621 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.423002 23220 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.427053 23220 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.416915 23219 consensus_queue.cc:237] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.405360 21669 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:58:59.392822 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.411931 23079 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:59.389695 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 71201.9us
I20250114 08:58:59.390070 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.394668 22896 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.409353 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.448307 21700 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:59.410823 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.411574 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.460090 21699 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:59.412204 21554 raft_consensus.cc:1270] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:59.412735 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:59.412649 21704 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:59.413921 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.461453 21552 raft_consensus.cc:2405] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:58:59.461849 21697 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:59.422618 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.396983 23164 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:59.430219 22239 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:58:59.404213 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:58:59.420027 22994 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.432375 23147 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.432766 23224 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:59.395352 23158 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.393260 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:58:59.434592 23155 raft_consensus.cc:3049] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:59.435221 23090 raft_consensus.cc:3049] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:58:59.437537 22881 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.423108 21452 catalog_manager.cc:5526] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 2, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:59.502207 21452 catalog_manager.cc:5526] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:59.440707 21553 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20250114 08:58:59.441198 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:59.430204 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.443374 21520 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:59.486711 23208 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:58:59.451849 21695 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:58:59.455579 21703 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:58:59.475987 21621 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:58:59.440506 23167 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:58:59.443933 21549 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:58:59.488919 23224 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.529578 23224 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.530287 23183 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:58:59.506322 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 63074.3us
W20250114 08:58:59.507974 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 77842.5us
I20250114 08:58:59.533883 21669 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.534852 23165 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.535661 21624 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250114 08:58:59.545665 21553 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 08:58:59.544037 21521 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.547875 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:59.508843 21452 catalog_manager.cc:5526] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 2, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:59.492062 23204 consensus_queue.cc:1046] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250114 08:58:59.522645 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.522768 23141 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.523818 23133 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.565940 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.490768 23147 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.525707 23158 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:58:59.550276 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.493921 23155 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:59.496879 23151 consensus_queue.cc:1035] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.121s
I20250114 08:58:59.582289 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:59.643383 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 58702.4us
I20250114 08:58:59.644735 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.652209 21548 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:58:59.653172 21627 raft_consensus.cc:1270] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 08:58:59.647796 23204 raft_consensus.cc:3049] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:58:59.567616 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:58:59.611788 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 69459us
I20250114 08:58:59.621161 21702 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.613150 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:59.566069 21697 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:58:59.672065 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.578727 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.616633 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:58:59.542126 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:58:59.609547 23165 consensus_queue.cc:1046] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250114 08:58:59.618031 21594 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:58:59.618245 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.613540 23071 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:58:59.519133 23149 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:59.519294 23213 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:59.633464 21624 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
W20250114 08:58:59.638649 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 89133.7us
I20250114 08:58:59.496529 23090 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:58:59.540107 23221 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.618592 23154 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:58:59.553093 23208 raft_consensus.cc:695] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:58:59.554797 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:59.611438 22837 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.612190 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:59.667429 21596 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:58:59.554404 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:58:59.657538 21452 catalog_manager.cc:5526] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 2 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:58:59.624661 23183 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.614987 23073 consensus_queue.cc:1046] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250114 08:58:59.743664 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:58:59.737270 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:58:59.738602 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 71259.6us
I20250114 08:58:59.746704 21627 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:59.739496 21626 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.729406 21629 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:58:59.742671 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.645200 23195 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:58:59.749053 21704 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:58:59.739230 21521 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.575170 23232 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.575412 22891 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:59.668134 23204 raft_consensus.cc:738] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:59.743331 21452 catalog_manager.cc:5526] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:58:59.743057 21697 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.754349 21596 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:59.748879 23241 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20250114 08:58:59.677047 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.754346 21521 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.794633 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:58:59.795333 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 245156us
I20250114 08:58:59.796332 21555 raft_consensus.cc:2363] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:58:59.817371 21704 raft_consensus.cc:2388] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:59.818018 23204 consensus_queue.cc:260] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.818497 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.818805 23221 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.819506 21704 raft_consensus.cc:2388] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 2: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:59.819655 23221 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:59.798696 21625 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:58:59.820114 23221 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:59.751084 23073 consensus_queue.cc:1225] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:58:59.792615 21553 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:58:59.675459 23233 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.738046 23213 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:58:59.738638 23079 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.644522 23235 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.176s
W20250114 08:58:59.817409 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 63153.8us
W20250114 08:58:59.795718 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 177766us
I20250114 08:58:59.801252 21452 catalog_manager.cc:5526] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 reported cstate change: term changed from 1 to 2, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 2 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } }
I20250114 08:58:59.840806 23074 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.851442 21521 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:58:59.852777 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:58:59.853566 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 47200.2us
I20250114 08:58:59.854039 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.813313 23175 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.825644 21520 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:59.827589 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 80649.3us
I20250114 08:58:59.831794 21697 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:58:59.864614 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.865224 21697 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:58:59.829093 23221 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:58:59.833376 23239 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20250114 08:58:59.821301 23108 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:59.804710 22885 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:58:59.729254 23151 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20250114 08:58:59.769209 23071 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.842980 21626 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:58:59.859498 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:58:59.916373 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 56975us
I20250114 08:58:59.749073 23155 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.750134 22837 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.820809 23101 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:59.821043 23052 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
W20250114 08:58:59.870137 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 44566.3us
I20250114 08:58:59.844262 23213 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.844059 22768 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:58:59.889068 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.912778 21697 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:58:59.799717 23232 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:58:59.889055 23157 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:58:59.859230 22994 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.918741 23156 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.889842 23079 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
W20250114 08:58:59.939209 23153 consensus_meta.cc:287] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.631s user 0.006s sys 0.006s
W20250114 08:58:59.959270 23230 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.554s user 0.002s sys 0.000s
I20250114 08:58:59.936528 23175 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:58:59.918078 22831 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250114 08:58:59.935981 23101 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.962108 22837 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.963181 23101 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.941365 23052 raft_consensus.cc:695] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:59.867071 23073 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:58:59.861227 23195 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:58:59.867069 23054 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:58:59.860232 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 318193us
I20250114 08:58:59.813901 22891 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.937851 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.976676 21697 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:58:59.941776 23141 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.846619 23233 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.914835 22885 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:59.937728 23083 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250114 08:58:59.994088 23211 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.994313 21695 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:58:59.994716 23211 raft_consensus.cc:3049] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:58:59.995162 23211 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:58:59.995585 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.996758 21695 raft_consensus.cc:2388] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:58:59.998132 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:58:59.999068 21695 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:59:00.000351 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:00.001405 22885 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:00.002769 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:00.004845 23243 log.cc:926] Time spent T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.253s user 0.000s sys 0.000s
I20250114 08:58:59.922461 23247 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:58:59.891129 23224 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.938128 22925 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:58:59.977792 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:00.006026 21697 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.824417 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:00.007050 21699 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:00.007445 21700 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:58:59.921586 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:58:59.914659 23108 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.947309 22843 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.011833 22861 raft_consensus.cc:3049] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:00.012230 22861 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:00.013315 22861 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.947906 23157 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.963474 22837 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:58:59.884771 23208 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:58:59.942086 22768 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.021832 22861 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.990646 23233 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:58:59.992285 23232 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
W20250114 08:58:59.992973 23249 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
W20250114 08:59:00.001060 23242 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250114 08:58:59.984419 23054 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.975590 23195 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:58:59.948904 23155 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:00.016386 21701 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:00.085914 21701 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:58:59.943936 23133 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:00.009307 22843 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.090169 22843 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:00.025374 22925 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:00.091042 22925 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:58:59.964851 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:00.097227 21704 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:00.098666 21704 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:00.028077 21699 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 08:59:00.099911 21699 pending_rounds.cc:77] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:59:00.100220 21699 pending_rounds.cc:99] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:00.100535 21699 raft_consensus.cc:2883] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:00.102473 21699 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:59:00.104084 21699 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 08:58:59.981774 23052 consensus_queue.cc:237] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:58:59.951092 23147 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:00.110302 23156 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.110961 23156 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:00.044037 21697 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:00.009320 23090 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.118398 23090 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:00.136811 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:00.139863 23156 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:00.104454 21699 pending_rounds.cc:77] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:59:00.150605 21699 pending_rounds.cc:99] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:00.151041 21699 raft_consensus.cc:2883] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:00.166945 21695 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:59:00.071030 23204 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:00.078524 23181 consensus_meta.cc:287] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.704s user 0.009s sys 0.005s
I20250114 08:58:59.996033 23211 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.165433 23195 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:59:00.209074 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 360748us
W20250114 08:59:00.287262 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 422218us
W20250114 08:59:00.212502 23164 consensus_meta.cc:287] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.726s user 0.011s sys 0.000s
W20250114 08:59:00.206459 23206 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.011s sys 0.010s
I20250114 08:59:00.305497 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:00.208580 23224 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:00.315917 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.209618 23157 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:00.319568 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.330308 21596 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:00.332746 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:00.334471 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:00.335769 21596 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:00.336040 21697 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:00.339303 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:00.339931 21548 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:00.341967 21703 raft_consensus.cc:1212] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20250114 08:59:00.342659 21703 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250114 08:59:00.372844 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 449253us
W20250114 08:59:00.447367 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 111677us
I20250114 08:59:00.448752 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:00.448966 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:00.450402 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:00.298269 23175 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.347822 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:00.457536 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:00.458194 21695 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:00.458634 21625 raft_consensus.cc:2388] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:00.458561 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:00.347918 22843 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:00.339206 23157 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:00.459502 22843 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.461635 22843 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:00.339047 23158 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
I20250114 08:59:00.465060 23149 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:00.465205 23158 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:00.465551 23149 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:00.465956 23149 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:00.469893 21702 consensus_meta.cc:287] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.724s user 0.003s sys 0.007s
I20250114 08:59:00.470293 21702 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:00.348713 23152 consensus_queue.cc:1046] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
I20250114 08:59:00.472096 23152 raft_consensus.cc:3049] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:00.472615 23183 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.307072 23181 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.474730 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:00.481019 21521 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:00.322705 23206 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.011s sys 0.010s
I20250114 08:59:00.320421 23164 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:00.492668 21626 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 08:59:00.505151 21521 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:00.328210 23047 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.509923 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:00.510542 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.511412 21551 raft_consensus.cc:2405] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250114 08:59:00.343217 22891 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:00.511162 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.350983 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:00.515434 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:00.587280 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:00.592602 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 116245us
I20250114 08:59:00.594635 21624 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:00.456346 23108 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:00.597270 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 92264.1us
I20250114 08:59:00.687548 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:00.688787 21521 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:00.343430 22768 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:00.604809 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:00.622871 21627 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:00.505468 23074 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:00.692183 23074 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:00.692651 23254 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20250114 08:59:00.694130 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1147) took 1076 ms. Trace:
I20250114 08:59:00.694767 21628 rpcz_store.cc:276] 0114 08:58:59.617530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.740242 (+122712us) service_pool.cc:225] Handling call
0114 08:58:59.740621 (+ 379us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:58:59.795685 (+ 55064us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:59.795730 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:58:59.820911 (+ 25181us) log.cc:844] Serialized 52 byte log entry
0114 08:59:00.073186 (+252275us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:00.073226 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:00.073294 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:00.693903 (+620609us) raft_consensus.cc:1680] finished
0114 08:59:00.693927 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:00.694068 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26937,"raft.queue_time_us":55283,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":12971,"thread_start_us":235937,"threads_started":1,"wal-append.queue_time_us":449902}
I20250114 08:59:00.503970 23181 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.698741 23181 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:00.699301 23181 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:00.699707 23181 raft_consensus.cc:695] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:00.495523 23233 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.186s
I20250114 08:59:00.701752 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.702986 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:00.703881 23248 log.cc:926] Time spent T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.404s user 0.001s sys 0.000s
W20250114 08:59:00.705140 21704 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.004s sys 0.013s
I20250114 08:59:00.472431 23152 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:00.491171 23101 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.473119 23183 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.704303 21551 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:00.745293 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 64685.1us
I20250114 08:59:00.513937 22843 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:00.509847 23157 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:00.514946 23073 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
W20250114 08:59:00.747951 21704 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.004s sys 0.017s
W20250114 08:59:00.747082 23220 consensus_meta.cc:287] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.319s user 0.011s sys 0.007s
I20250114 08:59:00.705399 23225 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.806620 23054 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:00.820838 23225 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.823114 23225 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:00.597524 23224 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.690284 23255 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:00.597208 23247 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.761093 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1194) took 1309 ms. Trace:
W20250114 08:59:00.761104 22881 consensus_meta.cc:287] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.257s user 0.005s sys 0.012s
W20250114 08:59:00.748281 23219 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.039s sys 0.092s
I20250114 08:59:00.692684 23074 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:59:00.758134 21553 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.927s user 0.013s sys 0.004s
W20250114 08:59:00.750669 23221 consensus_meta.cc:287] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.837s user 0.008s sys 0.001s
I20250114 08:59:00.700745 23181 consensus_queue.cc:237] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.808090 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1091) took 1243 ms. Trace:
I20250114 08:59:00.839066 21703 rpcz_store.cc:276] 0114 08:58:59.564915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.565065 (+ 150us) service_pool.cc:225] Handling call
0114 08:59:00.301850 (+736785us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:00.360509 (+ 58659us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:00.807816 (+447307us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b540023e6d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:00.808014 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1693548800}
I20250114 08:59:00.839835 21550 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 08:59:00.840243 21550 pending_rounds.cc:77] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 1
I20250114 08:59:00.801272 23259 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.806883 23257 consensus_queue.cc:1046] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20250114 08:59:00.771206 23052 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:00.820247 22925 consensus_meta.cc:287] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.729s user 0.009s sys 0.002s
I20250114 08:59:00.776016 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.858817 21551 raft_consensus.cc:2405] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250114 08:59:00.518438 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 184770us
I20250114 08:59:00.824915 21554 rpcz_store.cc:276] 0114 08:58:59.438665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.461360 (+ 22695us) service_pool.cc:225] Handling call
0114 08:58:59.487258 (+ 25898us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:59.491113 (+ 3855us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:59.491144 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:59.491563 (+ 419us) log.cc:844] Serialized 26 byte log entry
0114 08:58:59.931713 (+440150us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:58:59.931779 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:58:59.931880 (+ 101us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:00.748170 (+816290us) raft_consensus.cc:1680] finished
0114 08:59:00.748191 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:00.748351 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":182,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":6981,"thread_start_us":389529,"threads_started":1,"wal-append.queue_time_us":695335}
I20250114 08:59:00.518987 23047 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:00.833521 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 497806us
I20250114 08:59:00.887485 21669 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:00.887866 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.840488 21550 pending_rounds.cc:99] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:00.889189 21550 raft_consensus.cc:2883] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:00.823868 23224 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:00.832695 23219 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.984s user 0.039s sys 0.093s
I20250114 08:59:00.838289 21553 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:00.894009 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1198) took 1340 ms. Trace:
W20250114 08:59:00.759124 23253 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.267s user 0.003s sys 0.007s
W20250114 08:59:00.818571 23213 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.877s user 0.011s sys 0.000s
W20250114 08:59:00.801680 23250 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250114 08:59:00.506700 23175 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:00.911000 21702 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250114 08:59:00.850607 21668 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.041s sys 0.039s
W20250114 08:59:00.912424 21668 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.705s user 0.041s sys 0.039s
W20250114 08:59:00.912817 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 706077us
I20250114 08:59:00.686336 23155 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:00.894773 21551 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:00.894537 21553 rpcz_store.cc:276] 0114 08:58:59.553564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.616224 (+ 62660us) service_pool.cc:225] Handling call
0114 08:59:00.893925 (+1277701us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:00.849632 23157 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20250114 08:59:00.886821 23147 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:00.909862 21703 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250114 08:59:00.849447 23221 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.760680 23051 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.153s
I20250114 08:59:00.930814 22768 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.849385 22843 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20250114 08:59:00.895244 21549 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:00.936219 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.958086 21553 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
W20250114 08:59:00.939256 21668 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.565s (SENT) [suppressed 34 similar messages]
W20250114 08:59:00.961912 21668 leader_election.cc:336] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.766s (SENT)
W20250114 08:59:00.963011 21668 leader_election.cc:336] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.836s (SENT)
I20250114 08:59:00.963444 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:00.909389 23165 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:00.844590 22831 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.858460 23225 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20250114 08:59:00.949102 23260 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:00.951596 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1101) took 1356 ms. Trace:
I20250114 08:59:00.953316 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1188) took 1535 ms. Trace:
W20250114 08:59:00.966897 21668 leader_election.cc:336] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.926s (SENT)
I20250114 08:59:00.906675 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.838604 23247 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.975463 21700 rpcz_store.cc:276] 0114 08:58:59.595216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.008129 (+412913us) service_pool.cc:225] Handling call
0114 08:59:00.008559 (+ 430us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:00.008923 (+ 364us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:00.008953 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:00.009429 (+ 476us) log.cc:844] Serialized 26 byte log entry
0114 08:59:00.339651 (+330222us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:00.339688 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:00.339749 (+ 61us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:00.951380 (+611631us) raft_consensus.cc:1680] finished
0114 08:59:00.951406 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:00.951536 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4188,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":115,"thread_start_us":329875,"threads_started":1,"wal-append.queue_time_us":930302}
I20250114 08:59:00.861758 22925 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.861179 23052 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.937791 23155 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:00.977376 21554 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:59:00.944875 23256 raft_consensus.cc:3049] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:00.977212 21550 rpcz_store.cc:276] 0114 08:58:59.417729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.417855 (+ 126us) service_pool.cc:225] Handling call
0114 08:58:59.418129 (+ 274us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:00.889789 (+1471660us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:00.952997 (+ 63208us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001176d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:00.953240 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3268710272}
I20250114 08:59:01.116568 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:01.033965 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 67103.4us
W20250114 08:59:01.040190 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 350339us
W20250114 08:59:01.045184 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 561772us
I20250114 08:59:01.046921 23255 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:00.772096 23101 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:00.852591 23259 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.044162 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1380) took 1181 ms. Trace:
I20250114 08:59:01.136247 21704 rpcz_store.cc:276] 0114 08:58:59.862966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.115459 (+252493us) service_pool.cc:225] Handling call
0114 08:59:00.115866 (+ 407us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:00.116131 (+ 265us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:00.116166 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:00.162407 (+ 46241us) log.cc:844] Serialized 26 byte log entry
0114 08:59:00.816757 (+654350us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:00.816801 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:00.816876 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:01.043917 (+227041us) raft_consensus.cc:1680] finished
0114 08:59:01.043937 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:01.044097 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":653896,"threads_started":1,"wal-append.queue_time_us":879393}
I20250114 08:59:00.918845 23213 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.137825 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:00.968075 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:00.864470 23152 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.067054 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1377) took 1579 ms. Trace:
I20250114 08:59:00.979372 23165 raft_consensus.cc:695] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:00.976263 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 70973.3us
I20250114 08:59:01.050930 23264 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.148336 23101 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.050578 23258 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:00.908756 23149 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:01.085106 23267 log.cc:926] Time spent T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20250114 08:59:00.966001 23147 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.036900 23151 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.977726 22831 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.104175 22843 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:00.980069 21552 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
W20250114 08:59:01.112916 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 340182us
I20250114 08:59:01.113646 23284 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:01.090349 23183 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.031705 23260 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.117395 21550 raft_consensus.cc:2371] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4. Re-sending same reply.
I20250114 08:59:01.226528 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:01.227669 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.127285 21594 leader_election.cc:400] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:59:01.129374 21668 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:01.107393 23090 consensus_meta.cc:287] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.988s user 0.013s sys 0.000s
I20250114 08:59:01.138870 21554 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 1.
I20250114 08:59:01.228278 21594 leader_election.cc:403] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 08:59:01.141127 21696 rpcz_store.cc:276] 0114 08:58:59.488035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.488155 (+ 120us) service_pool.cc:225] Handling call
0114 08:58:59.501514 (+ 13359us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:58:59.507509 (+ 5995us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:58:59.507539 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:58:59.522041 (+ 14502us) log.cc:844] Serialized 26 byte log entry
0114 08:59:00.079603 (+557562us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:00.079658 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:00.079757 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:01.066664 (+986907us) raft_consensus.cc:1680] finished
0114 08:59:01.066695 (+ 31us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:01.066966 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5779,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":13025,"thread_start_us":419554,"threads_started":1,"wal-append.queue_time_us":690427}
W20250114 08:59:01.228871 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 99586.3us
I20250114 08:59:01.143610 21626 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250114 08:59:01.230116 21668 leader_election.cc:336] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.200s (SENT)
I20250114 08:59:01.146301 23247 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:01.146845 22885 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.145s user 0.011s sys 0.001s
I20250114 08:59:01.147109 22925 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.114190 23073 raft_consensus.cc:695] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:01.242597 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:01.243784 21554 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:01.150533 21521 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:01.154443 23283 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.154589 23278 consensus_queue.cc:1046] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:01.155907 23157 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:01.155786 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:01.165127 23263 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20250114 08:59:01.167508 23259 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:01.181619 23175 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.182880 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.185504 23052 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.115725 23256 raft_consensus.cc:738] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:01.192782 23224 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:01.195559 23204 consensus_meta.cc:287] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.011s user 0.010s sys 0.000s
W20250114 08:59:01.199957 22861 consensus_meta.cc:287] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.125s user 0.009s sys 0.000s
I20250114 08:59:01.149101 23047 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.202651 23152 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:00.972190 22768 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.125981 23219 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.126067 23270 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.125977 23275 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.148535 23213 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.223781 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.303081 21697 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:01.230510 21668 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:01.214005 23208 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.024s sys 0.039s
I20250114 08:59:01.211558 23206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57308 (request call id 453) took 3933 ms. Trace:
I20250114 08:59:01.235989 21548 raft_consensus.cc:2405] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20250114 08:59:01.115988 23155 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:59:01.245692 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 95256.1us
I20250114 08:59:01.312304 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.250385 21629 raft_consensus.cc:1270] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:59:01.250417 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.251847 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.253605 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.314810 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.254838 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.315548 21629 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:01.315655 21623 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:59:01.258399 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 131275us
I20250114 08:59:01.259474 21550 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:01.317989 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.243542 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.217126 23221 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.235097 23258 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:01.262993 23257 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.320678 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.325536 21629 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:01.326512 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.327055 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.326512 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.214461 23151 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.275980 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:01.308070 23208 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s user 0.024s sys 0.039s
I20250114 08:59:01.288801 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:01.266706 23054 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.219476 22843 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.309515 22837 raft_consensus.cc:3049] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:01.305051 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.360612 21554 raft_consensus.cc:2388] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:01.305452 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 75366.3us
W20250114 08:59:01.361265 21668 leader_election.cc:336] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.205s (SENT)
I20250114 08:59:01.361342 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.362596 21668 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:01.304379 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.308346 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.268437 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.366870 21551 raft_consensus.cc:2405] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:01.250689 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.445806 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.260818 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.373585 21626 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:01.287357 22861 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.267021 23271 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:01.272022 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:01.344923 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.456427 21629 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:01.457664 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.457779 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.458595 21629 raft_consensus.cc:2371] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4. Re-sending same reply.
I20250114 08:59:01.459726 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.345891 21701 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:01.445876 21699 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:01.461269 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.225646 22241 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:01.265669 22925 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:01.270053 23276 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.276641 21628 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:59:01.478750 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.287276 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:01.347441 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 116755us
I20250114 08:59:01.290378 23149 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:01.286721 22768 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:01.242378 23195 consensus_meta.cc:287] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.034s user 0.007s sys 0.006s
I20250114 08:59:01.354087 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.272363 23157 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.272800 21546 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:59:01.500231 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.445590 23052 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.359176 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.240890 22885 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.240088 23247 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.240260 23266 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20250114 08:59:01.364846 21697 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:01.218509 23284 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.264991 23090 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20250114 08:59:01.313654 23278 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.312840 22896 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:59:01.462816 21668 leader_election.cc:336] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.000s (SENT)
W20250114 08:59:01.527645 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 64896.9us
I20250114 08:59:01.216939 23264 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.287212 23219 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.481464 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.533010 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1162) took 1077 ms. Trace:
I20250114 08:59:01.287770 23275 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.490615 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.538058 21628 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:01.539013 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.539762 21628 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:01.357779 23213 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:01.492134 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 147479us
I20250114 08:59:01.493589 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.541033 21621 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:01.540846 23213 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.541968 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.542646 21621 raft_consensus.cc:2363] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4.
I20250114 08:59:01.542958 23213 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:01.543408 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.360158 22837 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:01.497783 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:01.544579 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:01.264039 23280 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.234s
I20250114 08:59:01.516508 23208 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:01.480525 22861 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.518066 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:01.448665 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:01.360436 23054 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.290887 23255 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.478606 23224 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:01.347436 23074 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.319134 23206 rpcz_store.cc:276] 0114 08:58:57.207205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.207329 (+ 124us) service_pool.cc:225] Handling call
0114 08:58:57.364991 (+157662us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b54000bcd20. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f55da94 00007fa63f5c8c47 00007fa63f5d4084 00007fa63f5d3743 00007fa63f5cbc1a 00007fa64452c6c9 00007fa63dbfc194 00007fa63dbfc122 00007fa63dbfc051 00007fa63dbfbfcc
0114 08:58:57.634689 (+269698us) log.cc:844] Serialized 508 byte log entry
0114 08:58:57.846406 (+211717us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b18000723a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c60c03a 00007fa645ac8be9 00007fa63c4cd930 00007fa63c691a14 00007fa63c691305 00007fa63f547580 00007fa63f5ad113 00007fa63f5d37e2 00007fa63f5cbc1a
0114 08:58:58.222228 (+375822us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b54000bcd20. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f55ecaf 00007fa63f5477a5 00007fa63f54c9d1 00007fa63f54c959 00007fa63f54c8e9 00007fa63f54c8b1 00007fa63f54ba7d 00007fa6456d1234 00007fa6456d1069 00007fa63c694866
0114 08:59:01.140865 (+2918637us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106754,"raft.queue_time_us":23002,"raft.run_cpu_time_us":4681,"raft.run_wall_time_us":453373,"spinlock_wait_cycles":16598528,"thread_start_us":192161,"threads_started":1,"wal-append.queue_time_us":329293,"wal-append.run_cpu_time_us":2951,"wal-append.run_wall_time_us":1321372}
I20250114 08:59:01.533399 21554 rpcz_store.cc:276] 0114 08:59:00.455802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.382104 (+926302us) service_pool.cc:225] Handling call
0114 08:59:01.532947 (+150843us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:01.578884 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.277684 23256 consensus_queue.cc:260] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.487613 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:01.311992 23165 consensus_queue.cc:237] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.525651 21695 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:01.546787 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.547613 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1176) took 1079 ms. Trace:
I20250114 08:59:01.581573 21550 rpcz_store.cc:276] 0114 08:59:00.468691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.227410 (+758719us) service_pool.cc:225] Handling call
0114 08:59:01.547460 (+320050us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b540024f4d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:01.547559 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71996800}
I20250114 08:59:01.548978 21696 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:01.547847 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:01.519536 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1167) took 1049 ms. Trace:
I20250114 08:59:01.554070 21551 raft_consensus.cc:2405] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250114 08:59:01.287485 23270 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.358024 21623 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:01.722522 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:01.723289 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.555619 23213 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.544608 22837 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:01.558322 23291 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250114 08:59:01.559459 23284 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:01.561452 21594 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:01.561898 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1206) took 1237 ms. Trace:
I20250114 08:59:01.292122 23283 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.534155 23278 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:01.751426 23282 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20250114 08:59:01.751904 23283 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.509958 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.752655 21624 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:01.752818 23157 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:01.753532 23290 log.cc:926] Time spent T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
W20250114 08:59:01.754973 23287 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.425s user 0.000s sys 0.000s
I20250114 08:59:01.570497 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.761253 21700 raft_consensus.cc:2405] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250114 08:59:01.572129 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:01.498059 23195 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.340735 23257 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.454412 23073 consensus_queue.cc:237] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.582654 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:01.366364 23273 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.161s
I20250114 08:59:01.775604 21555 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:01.777607 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1199) took 1465 ms. Trace:
I20250114 08:59:01.778090 21555 rpcz_store.cc:276] 0114 08:59:00.312127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.312262 (+ 135us) service_pool.cc:225] Handling call
0114 08:59:00.312640 (+ 378us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:01.776229 (+1463589us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:01.777280 (+ 1051us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540031cc54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:01.777525 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3363848576}
I20250114 08:59:01.745520 23275 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.766784 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 108) took 1302 ms. Trace:
I20250114 08:59:01.786842 21625 rpcz_store.cc:276] 0114 08:59:00.464913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.465791 (+ 878us) service_pool.cc:225] Handling call
0114 08:59:00.466200 (+ 409us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:00.721055 (+254855us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:00.744503 (+ 23448us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:00.827093 (+ 82590us) log.cc:844] Serialized 52 byte log entry
0114 08:59:01.096795 (+269702us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:01.096857 (+ 62us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:01.096943 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:01.766446 (+669503us) raft_consensus.cc:1680] finished
0114 08:59:01.766478 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:01.766686 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104619,"raft.queue_time_us":159975,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5254,"thread_start_us":240368,"threads_started":1,"wal-append.queue_time_us":648255}
I20250114 08:59:01.584106 23306 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.585377 23181 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:01.586072 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1207) took 1238 ms. Trace:
I20250114 08:59:01.801482 21548 rpcz_store.cc:276] 0114 08:59:00.348109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.450276 (+102167us) service_pool.cc:225] Handling call
0114 08:59:01.585818 (+1135542us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b540031c754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:01.585990 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1487535872}
I20250114 08:59:01.587862 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.803398 21553 raft_consensus.cc:2405] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 08:59:01.698972 21698 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250114 08:59:01.803966 21698 pending_rounds.cc:77] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 2
I20250114 08:59:01.804275 21698 pending_rounds.cc:99] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:01.804608 21698 raft_consensus.cc:2883] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:01.520453 23052 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.581172 21628 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
W20250114 08:59:01.809909 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 231171us
I20250114 08:59:01.810132 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.810734 21628 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:01.546325 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 321362us
I20250114 08:59:01.597368 21546 rpcz_store.cc:276] 0114 08:59:00.461332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.155413 (+694081us) service_pool.cc:225] Handling call
0114 08:59:01.505997 (+350584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:01.495358 23276 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.530283 23247 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.728956 21630 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:01.834650 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:01.722234 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 133709us
I20250114 08:59:01.733019 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:01.833056 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 271677us
I20250114 08:59:01.594959 23224 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.580740 22270 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:01.733891 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3b369b35736c413ca069db061ed36bc5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 10)
I20250114 08:59:01.574343 22831 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.744657 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.745759 21552 rpcz_store.cc:276] 0114 08:59:00.328604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.328746 (+ 142us) service_pool.cc:225] Handling call
0114 08:59:00.451951 (+123205us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:01.216855 (+764904us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:01.561656 (+344801us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b54003a0a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:01.561840 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1213979520}
I20250114 08:59:01.763581 21624 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:01.598325 23147 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:01.770758 23195 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.722345 23232 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:01.734623 22891 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.730820 22896 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:01.800633 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:01.947113 21625 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:01.807144 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.811180 21521 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:01.810168 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1183) took 1222 ms. Trace:
I20250114 08:59:01.947922 21553 rpcz_store.cc:276] 0114 08:59:00.588065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.958944 (+370879us) service_pool.cc:225] Handling call
0114 08:59:01.587671 (+628727us) spinlock_profiling.cc:243] Waited 9.05 ms on lock 0x7b5800000f98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:01.810107 (+222436us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20815616}
I20250114 08:59:01.737318 22925 raft_consensus.cc:695] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:01.724014 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.952306 21628 raft_consensus.cc:2405] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:01.720996 23256 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:01.872401 21627 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:01.870625 21548 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:01.765519 23286 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:59:01.812134 23258 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:01.742650 23213 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:01.910528 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:02.035303 21668 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:01.912834 21439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 178950us
I20250114 08:59:01.919785 21623 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:59:01.920320 21520 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:01.922080 21596 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:01.934474 21629 raft_consensus.cc:2371] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4. Re-sending same reply.
W20250114 08:59:01.899550 23291 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20250114 08:59:01.936383 21699 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:01.875909 23101 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:01.944813 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.047971 21552 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:01.934535 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:01.950750 22891 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:01.757479 22768 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:01.953102 21628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:57318
I20250114 08:59:02.077603 21628 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:02.078140 21628 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:01.953583 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.079566 21555 raft_consensus.cc:2388] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:01.828902 23181 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:02.086449 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.087033 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:01.949669 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 67883.7us
I20250114 08:59:01.915028 23158 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:01.866863 23151 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:01.988351 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:02.044476 23259 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:01.916581 23300 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.922582 23255 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:01.903514 23264 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.034475 21625 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:59:02.104496 23074 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:01.872618 22843 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:02.045104 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 233968us
I20250114 08:59:02.040675 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1156) took 1586 ms. Trace:
I20250114 08:59:02.188586 21549 rpcz_store.cc:276] 0114 08:59:00.453881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.454028 (+ 147us) service_pool.cc:225] Handling call
0114 08:59:00.675830 (+221802us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b5800000f98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:00.895007 (+219177us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:00.957548 (+ 62541us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:01.955465 (+997917us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540031bd50. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:02.040589 (+ 85124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":560153216}
I20250114 08:59:02.190160 21549 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:02.190891 21549 raft_consensus.cc:604] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.053553 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:01.947722 23152 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.669s user 0.010s sys 0.002s
I20250114 08:59:01.917253 23260 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:01.940806 22885 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:02.073657 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 153462us
W20250114 08:59:02.074834 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 152878us
I20250114 08:59:01.932380 23047 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.138s
I20250114 08:59:02.197240 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:02.200745 23312 log.cc:926] Time spent T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 08:59:02.200970 23257 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:01.921089 23289 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.308s user 0.001s sys 0.000s
I20250114 08:59:02.202769 22891 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.203213 22270 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:01.810278 23306 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.176813 23300 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.205690 23306 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.087553 21552 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:02.205925 23309 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:02.206554 23309 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:02.207585 21550 raft_consensus.cc:2405] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:02.105562 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.209175 21555 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:01.946416 23083 consensus_queue.cc:1046] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.131s
I20250114 08:59:02.210800 22837 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.210963 23083 raft_consensus.cc:3049] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:02.211359 23083 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:02.212216 23083 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.213492 23083 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:02.213742 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:01.954727 21554 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:02.123708 23221 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:02.125116 23219 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.114867 23052 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.115480 23213 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.116444 23305 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:02.116684 23224 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.237330 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.118654 23270 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.034832 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:01.984016 22896 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:02.092621 23195 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:02.120646 23298 consensus_queue.cc:1046] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20250114 08:59:02.121295 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.120688 23158 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:02.121974 23304 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
I20250114 08:59:02.122484 22861 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.124598 21548 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:02.282392 21548 raft_consensus.cc:1270] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 08:59:02.282907 21548 pending_rounds.cc:77] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 1
I20250114 08:59:02.283226 21548 pending_rounds.cc:99] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:02.283583 21548 raft_consensus.cc:2883] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250114 08:59:02.125202 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 186829us
I20250114 08:59:02.095405 22768 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.219193 23221 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.297955 22768 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:01.946702 23232 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.119954 23277 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.194s
W20250114 08:59:02.123296 23155 consensus_meta.cc:287] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.780s user 0.011s sys 0.000s
I20250114 08:59:02.035552 22272 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:02.107550 22881 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:02.194132 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:02.191689 21551 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:02.195453 21520 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.128623 23275 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.195719 21596 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.213943 21549 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:02.113621 22241 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.108022 21553 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:02.350518 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1179) took 1880 ms. Trace:
I20250114 08:59:02.350916 21553 rpcz_store.cc:276] 0114 08:59:00.469726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.987934 (+1518208us) service_pool.cc:225] Handling call
0114 08:59:02.350454 (+362520us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.351970 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.113833 23247 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.216917 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 109) took 1750 ms. Trace:
I20250114 08:59:02.354140 21630 rpcz_store.cc:276] 0114 08:59:00.466928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.515379 (+ 48451us) service_pool.cc:225] Handling call
0114 08:59:00.515662 (+ 283us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:01.868833 (+1353171us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.216640 (+347807us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540022cc54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:02.216837 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2774166912}
I20250114 08:59:02.355059 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1174) took 1891 ms. Trace:
I20250114 08:59:02.355475 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:02.355433 21549 rpcz_store.cc:276] 0114 08:59:00.463574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.213185 (+1749611us) service_pool.cc:225] Handling call
0114 08:59:02.213544 (+ 359us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.352078 (+138534us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.355011 (+ 2933us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.128867 23054 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.210405 21555 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 08:59:02.356164 21630 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:02.356734 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1169) took 1894 ms. Trace:
I20250114 08:59:02.357101 21555 rpcz_store.cc:276] 0114 08:59:00.461960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.209855 (+1747895us) service_pool.cc:225] Handling call
0114 08:59:02.210193 (+ 338us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.356441 (+146248us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.356682 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.358035 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.359056 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:02.359426 21555 raft_consensus.cc:604] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.359627 21555 raft_consensus.cc:612] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:02.244715 21701 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:02.361187 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.361418 21555 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 08:59:02.221709 22239 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:02.361793 21701 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:02.246076 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.362754 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:02.121904 22925 consensus_queue.cc:237] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.266315 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.366858 21703 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.265818 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:02.245536 21628 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.286792 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1177) took 1815 ms. Trace:
I20250114 08:59:02.274230 22267 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
W20250114 08:59:02.292647 23314 log.cc:926] Time spent T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20250114 08:59:02.292892 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.294159 23158 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:02.378166 23308 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20250114 08:59:02.221745 23206 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.251685 23305 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:02.388917 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1244) took 1133 ms. Trace:
I20250114 08:59:02.389175 23305 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.390115 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57308 (request call id 454) took 3501 ms. Trace:
W20250114 08:59:02.315083 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 119699us
I20250114 08:59:02.390503 21622 rpcz_store.cc:276] 0114 08:58:58.888642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:59.159610 (+270968us) service_pool.cc:225] Handling call
0114 08:59:02.389517 (+3229907us) spinlock_profiling.cc:243] Waited 2.59 s on lock 0x7b54000c9254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f552d50 00007fa63f5d5f2a 00007fa64452fc5e 00007fa63dc0c817 00007fa63dc0c7a2 00007fa63dc0c6d1 00007fa63dc0c64c 00007fa63dc0b902 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:02.390028 (+ 511us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5949530368}
I20250114 08:59:02.391410 21520 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:02.392505 21520 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.392810 23305 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:02.393385 23305 raft_consensus.cc:695] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:02.257530 23271 consensus_meta.cc:287] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.749s user 0.011s sys 0.000s
I20250114 08:59:02.395074 23271 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:59:02.396508 21630 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:02.397053 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1116) took 1176 ms. Trace:
I20250114 08:59:02.397615 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.398828 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.399415 21630 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
W20250114 08:59:02.348984 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 153384us
I20250114 08:59:02.349318 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.401018 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.401384 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.402055 23271 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:02.402498 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.394253 23305 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:02.363219 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.403422 21697 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.404269 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.404858 21697 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:02.405332 21596 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.406096 21697 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:02.406637 21596 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:02.361778 21555 pending_rounds.cc:77] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 1
I20250114 08:59:02.363346 21701 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.302904 23181 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.299377 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:02.218206 23255 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.302592 23309 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.365109 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.441615 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.365660 21521 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:02.442581 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 76976.1us
I20250114 08:59:02.443816 21521 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:02.369439 21548 rpcz_store.cc:276] 0114 08:59:00.469037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.281423 (+1812386us) service_pool.cc:225] Handling call
0114 08:59:02.281889 (+ 466us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.284129 (+ 2240us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.284398 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.348811 23233 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.347127 22861 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.316834 23257 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:02.316987 23316 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.351585 22843 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:02.382171 21628 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.367281 23221 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.385172 23276 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.369432 23211 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:02.386507 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.386611 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.389357 21624 rpcz_store.cc:276] 0114 08:59:01.255833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.914843 (+659010us) service_pool.cc:225] Handling call
0114 08:59:01.915209 (+ 366us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:01.915422 (+ 213us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:01.915444 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:01.945628 (+ 30184us) log.cc:844] Serialized 26 byte log entry
0114 08:59:02.219783 (+274155us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:02.219816 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.220004 (+ 188us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:02.388702 (+168698us) raft_consensus.cc:1680] finished
0114 08:59:02.388723 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.388859 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":268021,"threads_started":1,"wal-append.queue_time_us":395538}
I20250114 08:59:02.473896 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.474792 21624 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:02.394557 21520 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:02.298847 23297 log.cc:926] Time spent T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250114 08:59:02.400225 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.124929 23101 raft_consensus.cc:695] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:02.480988 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 86478.9us
I20250114 08:59:02.407742 21555 pending_rounds.cc:99] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:02.481911 21555 raft_consensus.cc:2883] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:02.482154 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.482820 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1178) took 2013 ms. Trace:
I20250114 08:59:02.472072 21696 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.408267 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1251) took 1149 ms. Trace:
I20250114 08:59:02.483826 21629 rpcz_store.cc:276] 0114 08:59:01.258618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.459403 (+200785us) service_pool.cc:225] Handling call
0114 08:59:02.408035 (+948632us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b540022e054. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:02.408191 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1090026368}
I20250114 08:59:02.460588 22768 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.397409 21698 rpcz_store.cc:276] 0114 08:59:01.221041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.221188 (+ 147us) service_pool.cc:225] Handling call
0114 08:59:01.221494 (+ 306us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:01.805139 (+583645us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.396830 (+591691us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400327b54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:02.397000 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1096872832}
I20250114 08:59:02.475979 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.462908 23309 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.418561 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:02.354496 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:02.421903 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:02.487401 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 65630.3us
I20250114 08:59:02.488812 21596 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:02.380928 21546 raft_consensus.cc:2405] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:02.361511 23260 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:02.409171 21695 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.436436 23206 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:02.436908 23052 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.368184 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.519788 21553 raft_consensus.cc:2405] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 08:59:02.454887 21594 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.456835 21701 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.520485 21622 raft_consensus.cc:1270] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:59:02.487324 23309 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.460484 21669 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:02.527477 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 67104.8us
I20250114 08:59:02.461769 21700 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:59:02.524292 21622 raft_consensus.cc:1270] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:59:02.463135 21518 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
W20250114 08:59:02.530061 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 67007.4us
I20250114 08:59:02.317162 23155 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.466894 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.352957 23275 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.503686 21696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82452b55145444529bf0609cd7767adf"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
I20250114 08:59:02.425026 23271 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:02.477716 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:57318 (request call id 1245) took 1212 ms. Trace:
I20250114 08:59:02.469120 23257 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.459303 23277 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:02.411118 21626 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:02.483186 21555 rpcz_store.cc:276] 0114 08:59:00.469376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.360112 (+1890736us) service_pool.cc:225] Handling call
0114 08:59:02.360541 (+ 429us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.482561 (+122020us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.482749 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.491603 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.542156 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:02.464124 23233 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:02.497021 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 275375us
I20250114 08:59:02.460357 22861 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.524235 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.532414 21548 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:02.420769 23158 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:02.455428 21521 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:02.555405 21549 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:02.518218 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1189) took 1386 ms. Trace:
W20250114 08:59:02.558351 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 103011us
I20250114 08:59:02.534797 21696 raft_consensus.cc:604] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:02.559468 21696 raft_consensus.cc:612] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250114 08:59:02.537380 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 82656.6us
I20250114 08:59:02.560966 21594 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.561842 23271 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:59:02.562461 23318 log.cc:926] Time spent T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20250114 08:59:02.562661 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1266) took 1009 ms. Trace:
I20250114 08:59:02.563055 21623 rpcz_store.cc:276] 0114 08:59:01.552962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.036507 (+483545us) service_pool.cc:225] Handling call
0114 08:59:02.045765 (+ 9258us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:02.103706 (+ 57941us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.103767 (+ 61us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:02.126479 (+ 22712us) log.cc:844] Serialized 52 byte log entry
0114 08:59:02.351103 (+224624us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:02.351144 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.351211 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:02.562435 (+211224us) raft_consensus.cc:1680] finished
0114 08:59:02.562458 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.562603 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92812,"raft.queue_time_us":45655,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":29448,"thread_start_us":135465,"threads_started":1,"wal-append.queue_time_us":277140}
I20250114 08:59:02.564895 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:02.538650 23090 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.565358 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.565723 23090 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.566125 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:02.566646 23255 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.566736 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.567389 23232 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.567530 21548 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:02.568742 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 113) took 1004 ms. Trace:
I20250114 08:59:02.493187 22768 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.569977 23233 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.569262 21627 rpcz_store.cc:276] 0114 08:59:01.564204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.213933 (+649729us) service_pool.cc:225] Handling call
0114 08:59:02.214265 (+ 332us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.214595 (+ 330us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.214622 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:02.219531 (+ 4909us) log.cc:844] Serialized 26 byte log entry
0114 08:59:02.445293 (+225762us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:02.445336 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.445400 (+ 64us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:02.568532 (+123132us) raft_consensus.cc:1680] finished
0114 08:59:02.568553 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.568680 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":646,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":25376,"thread_start_us":178895,"threads_started":1,"wal-append.queue_time_us":250597}
I20250114 08:59:02.547574 23277 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.570676 23277 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.571024 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.571632 22768 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.536830 23280 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250114 08:59:02.572213 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.573467 23278 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.501230 23307 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:02.540022 23047 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:02.542979 21555 raft_consensus.cc:3049] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:59:02.501590 22837 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:02.518556 23224 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.590078 23224 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.591012 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1400) took 1019 ms. Trace:
I20250114 08:59:02.591421 21699 rpcz_store.cc:276] 0114 08:59:01.571305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.571876 (+ 571us) service_pool.cc:225] Handling call
0114 08:59:02.590836 (+1018960us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400346554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:02.590952 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":399241600}
I20250114 08:59:02.470665 23316 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.572654 23158 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.594117 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:02.594393 23316 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.594532 23158 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.496239 23276 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:02.595676 23276 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:02.595885 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1200) took 2282 ms. Trace:
I20250114 08:59:02.596150 23276 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.596349 21547 rpcz_store.cc:276] 0114 08:59:00.313215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:00.317593 (+ 4378us) service_pool.cc:225] Handling call
0114 08:59:00.805199 (+487606us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b5800000f98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:00.805458 (+ 259us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:01.140087 (+334629us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:01.141936 (+ 1849us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:01.258325 (+116389us) log.cc:844] Serialized 26 byte log entry
0114 08:59:01.721777 (+463452us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:01.729890 (+ 8113us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:01.747302 (+ 17412us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:02.575142 (+827840us) raft_consensus.cc:1680] finished
0114 08:59:02.576680 (+ 1538us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.595813 (+ 19133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28493,"raft.queue_time_us":234676,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":168,"spinlock_wait_cycles":37806464,"thread_start_us":556067,"threads_started":2,"wal-append.queue_time_us":576820}
I20250114 08:59:02.598230 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:02.518942 23052 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.553571 21553 raft_consensus.cc:2388] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:02.605314 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 46017.9us
I20250114 08:59:02.606216 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.607393 21553 raft_consensus.cc:2405] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 08:59:02.554440 23322 log.cc:926] Time spent T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250114 08:59:02.521660 23225 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:02.495791 23281 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:02.493055 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:02.368485 23208 consensus_queue.cc:1046] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20250114 08:59:02.554298 23219 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:02.535583 23309 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:02.534792 21628 rpcz_store.cc:276] 0114 08:59:01.256138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.952983 (+696845us) service_pool.cc:225] Handling call
0114 08:59:02.468485 (+515502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.562278 21594 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.537753 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.565410 23257 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.576983 21551 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
W20250114 08:59:02.542520 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 51045us
I20250114 08:59:02.577589 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.585225 21555 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:02.547358 23315 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:02.557925 21546 rpcz_store.cc:276] 0114 08:59:01.131841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.120984 (+989143us) service_pool.cc:225] Handling call
0114 08:59:02.501465 (+380481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.552178 23073 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:02.600749 21627 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.546900 22896 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.606963 21521 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:02.572291 23306 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.611742 23325 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.573815 22861 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.643616 23306 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.646118 23306 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:02.646415 23275 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.646176 23302 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
W20250114 08:59:02.621390 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 83711.3us
I20250114 08:59:02.642102 23281 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.648895 23281 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.650503 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:02.622606 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 60402.6us
I20250114 08:59:02.624558 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.624610 21548 raft_consensus.cc:2388] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:02.636101 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.652881 21628 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.619159 21553 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:02.641474 21521 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:02.611028 23052 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:02.598744 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:02.613943 23225 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.566154 23090 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:02.634505 23206 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:02.624243 23224 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:02.585551 23101 consensus_queue.cc:237] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:02.666935 21668 leader_election.cc:336] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.546s (SENT)
I20250114 08:59:02.859395 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1175) took 2395 ms. Trace:
I20250114 08:59:02.740489 22896 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.164s
I20250114 08:59:02.649266 23281 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:02.631260 21623 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
W20250114 08:59:02.704623 21626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1260) took 1336 ms (client timeout 1559 ms). Trace:
I20250114 08:59:02.875072 21554 rpcz_store.cc:276] 0114 08:59:00.468276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.214717 (+1746441us) service_pool.cc:225] Handling call
0114 08:59:02.215038 (+ 321us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.215253 (+ 215us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.215292 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:02.215700 (+ 408us) log.cc:844] Serialized 26 byte log entry
0114 08:59:02.497837 (+282137us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:02.497896 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.498037 (+ 141us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:02.859092 (+361055us) raft_consensus.cc:1680] finished
0114 08:59:02.859119 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.859310 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":281746,"threads_started":1,"wal-append.queue_time_us":606649}
I20250114 08:59:02.653764 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:02.653610 21548 raft_consensus.cc:1270] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:02.659703 21669 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:02.873580 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 221738us
I20250114 08:59:02.876785 21553 raft_consensus.cc:604] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:02.876039 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 251598us
W20250114 08:59:02.874795 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 207920us
I20250114 08:59:02.641064 23073 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:02.906344 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 264920us
I20250114 08:59:02.907954 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:02.908131 21626 rpcz_store.cc:269] 0114 08:59:01.368761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.457398 (+ 88637us) service_pool.cc:225] Handling call
0114 08:59:02.704372 (+1246974us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b540039ec54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:02.704545 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2155647872}
I20250114 08:59:02.908768 21553 raft_consensus.cc:612] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250114 08:59:02.909242 21550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1170) took 2447 ms (client timeout 2542 ms). Trace:
W20250114 08:59:02.909758 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 250138us
W20250114 08:59:02.909631 21550 rpcz_store.cc:269] 0114 08:59:00.462293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.582393 (+1120100us) service_pool.cc:225] Handling call
0114 08:59:02.909062 (+1326669us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b540031b854. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:02.909188 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":999002880}
I20250114 08:59:02.910662 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:02.911254 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1184) took 2314 ms. Trace:
I20250114 08:59:02.911608 21550 rpcz_store.cc:276] 0114 08:59:00.597248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.910496 (+2313248us) service_pool.cc:225] Handling call
0114 08:59:02.910797 (+ 301us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.910970 (+ 173us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.910996 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:02.911041 (+ 45us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:02.911066 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.911115 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.911206 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.912199 21623 raft_consensus.cc:1270] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:02.913365 21520 leader_election.cc:400] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4.
I20250114 08:59:02.914215 21520 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.915380 21626 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20250114 08:59:02.915722 21668 leader_election.cc:336] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.560s (SENT)
I20250114 08:59:02.915903 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.916172 21668 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:02.916939 21626 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:02.917116 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.919457 21550 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:02.920302 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1186) took 2237 ms. Trace:
I20250114 08:59:02.920991 21550 rpcz_store.cc:276] 0114 08:59:00.682804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.918745 (+2235941us) service_pool.cc:225] Handling call
0114 08:59:02.919091 (+ 346us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.920023 (+ 932us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.920239 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.915026 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1181) took 2444 ms. Trace:
I20250114 08:59:02.917934 21698 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:02.926769 21623 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:02.926509 21548 rpcz_store.cc:276] 0114 08:59:00.470441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.652990 (+2182549us) service_pool.cc:225] Handling call
0114 08:59:02.653303 (+ 313us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.914790 (+261487us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.914964 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.928392 21548 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:59:02.928534 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:02.929175 21698 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:02.755913 23074 raft_consensus.cc:3049] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:02.933012 21551 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.968037 21521 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:02.982877 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 92) took 1640 ms. Trace:
I20250114 08:59:02.983687 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.982888 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.985553 21702 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:02.946070 21594 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:02.986371 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.652199 23325 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:02.673074 23302 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:02.705044 23165 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:02.992031 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:02.762199 23083 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.947193 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1191) took 1814 ms. Trace:
I20250114 08:59:02.995039 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:02.940743 22896 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:03.014513 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.014550 22896 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.990379 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1168) took 2528 ms. Trace:
I20250114 08:59:02.983292 21549 rpcz_store.cc:276] 0114 08:59:01.342076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.982065 (+1639989us) service_pool.cc:225] Handling call
0114 08:59:02.982375 (+ 310us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.982560 (+ 185us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.982588 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:02.982631 (+ 43us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:02.982659 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.982720 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.982819 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:02.996409 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.018913 21552 rpcz_store.cc:276] 0114 08:59:00.461651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.206645 (+1744994us) service_pool.cc:225] Handling call
0114 08:59:02.207017 (+ 372us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.355836 (+148819us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.355866 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:02.375712 (+ 19846us) log.cc:844] Serialized 26 byte log entry
0114 08:59:02.532983 (+157271us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:02.576910 (+ 43927us) log.cc:844] Serialized 14 byte log entry
0114 08:59:02.577266 (+ 356us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.925379 (+348113us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540016f154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:02.925449 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:02.925515 (+ 66us) raft_consensus.cc:1680] finished
0114 08:59:02.990130 (+ 64615us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:02.990302 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1489,"prepare.queue_time_us":42943,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":15393,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":37062,"spinlock_wait_cycles":341385984,"thread_start_us":93853,"threads_started":1,"wal-append.queue_time_us":215850}
I20250114 08:59:03.021230 21696 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:02.902117 22885 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:02.997368 21700 raft_consensus.cc:1270] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:59:02.702764 21546 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 08:59:02.856420 23219 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:03.011126 21701 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:02.881275 23225 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:03.012769 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.742405 23278 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:02.657119 23257 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:03.015776 21702 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:02.998958 21555 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:02.988698 21553 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250114 08:59:03.045768 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 93) took 1703 ms. Trace:
I20250114 08:59:03.046221 21553 rpcz_store.cc:276] 0114 08:59:01.342422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.987901 (+1645479us) service_pool.cc:225] Handling call
0114 08:59:02.988249 (+ 348us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.045427 (+ 57178us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.045695 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:03.026755 23181 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.011s sys 0.021s
W20250114 08:59:03.017997 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 72047.9us
I20250114 08:59:03.053368 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.053843 21594 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:03.056319 21594 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:02.649968 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 536406us
I20250114 08:59:02.943230 23233 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:03.061658 23257 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.062201 23233 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:02.980801 23316 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:03.064730 23257 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:02.980588 23224 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:03.065078 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.941968 23074 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:03.071646 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.071863 23074 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.073062 23074 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:03.025159 23156 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 08:59:03.075894 21668 leader_election.cc:336] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.594s (SENT)
I20250114 08:59:02.996435 23298 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:02.711685 22768 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.302s
I20250114 08:59:02.981050 23315 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:02.703725 23307 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:03.029536 23329 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:03.003906 23232 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:03.008809 23302 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:02.753849 23300 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:02.962280 23221 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:03.110312 23221 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.040834 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:03.112953 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 72247.5us
I20250114 08:59:03.078567 23156 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:03.115327 21596 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:02.998729 23325 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.048454 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:02.938431 23073 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:02.765492 23277 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.047567 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.018719 22896 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:03.061079 21698 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:03.069532 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.131022 21700 raft_consensus.cc:2405] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:03.004124 23295 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20250114 08:59:03.080442 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:03.031337 23165 raft_consensus.cc:695] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:03.049949 22885 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:03.047648 23278 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.085961 23307 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:03.090538 23329 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:03.090996 23315 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.078285 23298 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:03.019459 23083 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.068437 23255 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:03.051107 22881 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:02.757230 23155 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.078956 23257 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:03.123454 21702 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
W20250114 08:59:03.053246 23181 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.011s sys 0.021s
I20250114 08:59:03.051435 23219 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.117350 23302 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.092098 23232 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.117231 23156 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.128573 23051 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:03.026459 21546 pending_rounds.cc:77] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 1
I20250114 08:59:03.016957 21548 rpcz_store.cc:276] 0114 08:59:01.132522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.927845 (+1795323us) service_pool.cc:225] Handling call
0114 08:59:02.928103 (+ 258us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.928811 (+ 708us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:02.947117 (+ 18306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:03.131431 21698 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:03.136801 21701 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:03.345492 23300 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 08:59:03.184845 23276 consensus_meta.cc:287] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.588s user 0.008s sys 0.002s
I20250114 08:59:03.348773 23276 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.178352 23277 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:03.180163 23306 consensus_meta.cc:287] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.534s user 0.007s sys 0.000s
I20250114 08:59:03.153718 23278 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.174716 23232 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:03.185073 23281 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.164142 23083 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.354743 21695 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
I20250114 08:59:03.354456 23306 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.162597 23298 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:59:03.258499 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 143120us
I20250114 08:59:03.246349 21546 pending_rounds.cc:99] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:03.164820 22881 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:03.381973 22881 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:03.147408 23165 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.376514 21695 raft_consensus.cc:604] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:03.176574 23051 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:03.384928 23051 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:03.199707 23302 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:03.385485 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 225551us
I20250114 08:59:03.388041 23302 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.388994 21550 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:03.185966 23090 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.389459 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.181922 23156 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:03.181175 23211 consensus_meta.cc:287] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.701s user 0.008s sys 0.003s
I20250114 08:59:03.391003 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:59:03.183080 23225 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.396667 23306 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:03.396889 21548 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:03.399319 21555 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:03.154667 23329 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.403671 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1190) took 2271 ms. Trace:
I20250114 08:59:03.405311 23276 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:03.165470 23257 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:03.426960 21549 raft_consensus.cc:1235] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:03.406736 23090 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:03.200095 23325 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.426893 21550 rpcz_store.cc:276] 0114 08:59:01.132161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.922300 (+1790139us) service_pool.cc:225] Handling call
0114 08:59:02.922626 (+ 326us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.402140 (+479514us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.403278 (+ 1138us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b540028f0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:03.403588 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1071361408}
I20250114 08:59:03.181337 23221 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.168139 23233 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:03.380982 21546 raft_consensus.cc:2883] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:03.434417 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1218) took 1058 ms. Trace:
I20250114 08:59:03.162405 23315 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:03.150090 22885 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.428797 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:03.184620 22896 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.170920 23219 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:03.445291 21548 rpcz_store.cc:276] 0114 08:59:02.369287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.395385 (+1026098us) service_pool.cc:225] Handling call
0114 08:59:03.396129 (+ 744us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.424442 (+ 28313us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.427558 (+ 3116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:03.481942 21548 raft_consensus.cc:1235] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:03.483009 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1223) took 1088 ms. Trace:
I20250114 08:59:03.483623 21548 rpcz_store.cc:276] 0114 08:59:02.394223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.480983 (+1086760us) service_pool.cc:225] Handling call
0114 08:59:03.481578 (+ 595us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.482660 (+ 1082us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.482919 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:03.185659 23073 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.527717 22267 raft_consensus.cc:1235] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:03.530987 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.147791 23052 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:03.536356 21622 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.002s sys 0.016s
I20250114 08:59:03.139273 23309 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.186504 23307 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:03.495117 23320 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.958s user 0.020s sys 0.045s
I20250114 08:59:03.559576 21624 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:03.548844 21622 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s user 0.002s sys 0.016s
I20250114 08:59:03.551235 21621 raft_consensus.cc:2405] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:03.555526 21547 raft_consensus.cc:2388] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:03.566082 21621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1261) took 2117 ms (client timeout 2183 ms). Trace:
I20250114 08:59:03.566582 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1219) took 1185 ms. Trace:
W20250114 08:59:03.566943 21625 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.026s sys 0.017s
I20250114 08:59:03.548722 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.567134 21547 rpcz_store.cc:276] 0114 08:59:02.381330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.598432 (+217102us) service_pool.cc:225] Handling call
0114 08:59:03.566157 (+967725us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b5400193a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:03.566466 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1548544256}
I20250114 08:59:03.567952 21553 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:03.553128 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 94) took 2210 ms. Trace:
I20250114 08:59:03.568924 21549 rpcz_store.cc:276] 0114 08:59:01.342726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.020323 (+1677597us) service_pool.cc:225] Handling call
0114 08:59:03.020734 (+ 411us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.446009 (+425275us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.526190 (+ 80181us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b540031ae54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:03.553005 (+ 26815us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":872673024}
I20250114 08:59:03.569586 21554 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:03.554446 23073 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.571854 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1182) took 3101 ms. Trace:
I20250114 08:59:03.569810 23183 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:03.569586 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1213) took 1280 ms. Trace:
I20250114 08:59:03.572937 21553 rpcz_store.cc:276] 0114 08:59:02.289154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.049155 (+760001us) service_pool.cc:225] Handling call
0114 08:59:03.049536 (+ 381us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:03.568514 (+518978us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.569240 (+ 726us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b5400194954. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:03.569521 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1176566528}
I20250114 08:59:03.573110 23183 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.552553 23257 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.552488 23156 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.578204 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.579313 21623 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:03.558691 23052 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.557893 23309 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.580942 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.582513 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:03.567339 21625 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.071s user 0.026s sys 0.017s
I20250114 08:59:03.583169 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1231) took 1146 ms. Trace:
I20250114 08:59:03.583307 21628 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:03.586962 21549 rpcz_store.cc:276] 0114 08:59:02.436752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.581690 (+1144938us) service_pool.cc:225] Handling call
0114 08:59:03.582342 (+ 652us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.582775 (+ 433us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:03.582812 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:03.582866 (+ 54us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:03.582925 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.582991 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:03.583102 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:03.544922 22240 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.016s sys 0.057s
W20250114 08:59:03.560657 21697 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.025s sys 0.063s
I20250114 08:59:03.572340 21554 rpcz_store.cc:276] 0114 08:59:00.512441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.942449 (+2430008us) service_pool.cc:225] Handling call
0114 08:59:03.569276 (+626827us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.570366 (+ 1090us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.571517 (+ 1151us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400193a50. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:03.571762 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1437289216}
W20250114 08:59:03.549028 23266 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.019s sys 0.043s
W20250114 08:59:03.605110 22240 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.016s sys 0.057s
W20250114 08:59:03.566625 21621 rpcz_store.cc:269] 0114 08:59:01.448812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:01.543155 (+ 94343us) service_pool.cc:225] Handling call
0114 08:59:03.565749 (+2022594us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54003babd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:03.565973 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4488686848}
W20250114 08:59:03.610352 21697 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.114s user 0.026s sys 0.063s
W20250114 08:59:03.597644 23320 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 1.059s user 0.021s sys 0.046s
I20250114 08:59:03.555212 23325 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.558864 23281 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:03.645972 21520 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.000s sys 0.016s
W20250114 08:59:03.599817 23266 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.048s user 0.019s sys 0.044s
I20250114 08:59:03.613734 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.652067 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.662092 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:03.675825 21520 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.693s user 0.002s sys 0.018s
W20250114 08:59:03.675925 21704 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.004s sys 0.028s
W20250114 08:59:03.704715 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 1.12249e+06us
W20250114 08:59:03.708525 21704 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.005s sys 0.028s
W20250114 08:59:03.710871 21521 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.016s sys 0.043s
W20250114 08:59:03.713591 21630 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s user 0.001s sys 0.048s
W20250114 08:59:03.721357 21519 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.009s sys 0.012s
I20250114 08:59:03.718580 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:03.721421 21521 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.017s sys 0.044s
W20250114 08:59:03.731509 21519 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.754s user 0.009s sys 0.013s
W20250114 08:59:03.730549 21630 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s user 0.001s sys 0.048s
I20250114 08:59:03.728430 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1180) took 3250 ms. Trace:
I20250114 08:59:03.732592 21628 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
W20250114 08:59:03.716320 22843 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.023s sys 0.031s
I20250114 08:59:03.737931 21546 rpcz_store.cc:276] 0114 08:59:00.470083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.645165 (+2175082us) service_pool.cc:225] Handling call
0114 08:59:02.702237 (+ 57072us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.611903 (+909666us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.720215 (+108312us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:03.743691 23208 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.013s sys 0.012s
W20250114 08:59:03.743997 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 161543us
W20250114 08:59:03.744235 23208 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.103s user 0.014s sys 0.012s
W20250114 08:59:03.745981 21703 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.014s sys 0.012s
I20250114 08:59:03.744885 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:03.748653 21696 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.005s sys 0.022s
W20250114 08:59:03.749166 21696 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.006s sys 0.022s
I20250114 08:59:03.749209 23256 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:03.749454 23149 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:03.746129 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:03.753962 22925 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.003s sys 0.008s
W20250114 08:59:03.754071 23305 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.011s sys 0.012s
W20250114 08:59:03.745560 22843 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.134s user 0.024s sys 0.031s
I20250114 08:59:03.759307 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.762595 23307 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:03.763553 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 849305us
I20250114 08:59:03.763960 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.766224 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.766336 21520 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:03.767313 22861 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.013s user 0.003s sys 0.029s
W20250114 08:59:03.767979 22861 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.062s user 0.003s sys 0.029s
W20250114 08:59:03.768067 21699 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.004s sys 0.018s
W20250114 08:59:03.768554 23101 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.016s sys 0.012s
I20250114 08:59:03.768739 23211 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:03.767433 21626 raft_consensus.cc:2388] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:03.771936 21703 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.015s sys 0.013s
W20250114 08:59:03.769992 22925 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.948s user 0.004s sys 0.008s
I20250114 08:59:03.771169 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1137) took 1356 ms. Trace:
W20250114 08:59:03.754004 22241 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.006s sys 0.029s
W20250114 08:59:03.781091 21699 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s user 0.005s sys 0.018s
W20250114 08:59:03.770994 23305 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s user 0.011s sys 0.013s
I20250114 08:59:03.787779 21623 raft_consensus.cc:2388] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:03.778040 23101 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.017s sys 0.012s
W20250114 08:59:03.789392 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 813561us
I20250114 08:59:03.791540 22896 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:03.797712 21628 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
W20250114 08:59:03.802600 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 286764us
I20250114 08:59:03.804443 21596 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:03.794736 22241 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.006s sys 0.029s
I20250114 08:59:03.807353 22925 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:59:03.808301 22266 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.805101 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:03.811795 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:03.812530 21555 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:03.776446 21696 raft_consensus.cc:2949] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:03.774631 23047 consensus_queue.cc:1046] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20250114 08:59:03.815182 21552 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
W20250114 08:59:03.861058 21668 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.023s sys 0.046s
I20250114 08:59:03.798456 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1228) took 1402 ms. Trace:
W20250114 08:59:03.865792 21551 consensus_meta.cc:287] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.919s user 0.008s sys 0.009s
W20250114 08:59:03.865422 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 46647.5us
I20250114 08:59:03.878284 23156 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:03.878911 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:03.886088 21521 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:03.872085 21668 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.745s user 0.023s sys 0.048s
W20250114 08:59:03.894310 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 776584us
I20250114 08:59:03.897029 23307 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:03.897555 23307 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:03.897816 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 781904us
I20250114 08:59:03.898358 23307 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:03.897723 23225 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:03.897607 21521 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:03.905398 23358 consensus_queue.cc:1035] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:03.908058 23309 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:03.887291 23183 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:03.861078 23361 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.053s user 0.002s sys 0.000s
I20250114 08:59:03.889075 23074 raft_consensus.cc:3049] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:03.865265 21627 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:03.890275 23257 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:03.866225 21555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1235) took 1412 ms (client timeout 1759 ms). Trace:
I20250114 08:59:03.891264 21551 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:03.910965 21700 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 1. Current term is 2. Ops: []
I20250114 08:59:03.776163 21697 rpcz_store.cc:276] 0114 08:59:02.401609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.429948 (+ 28339us) service_pool.cc:225] Handling call
0114 08:59:02.430227 (+ 279us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.430409 (+ 182us) raft_consensus.cc:1529] Early marking committed up to index 5
0114 08:59:02.495677 (+ 65268us) log.cc:844] Serialized 14 byte log entry
0114 08:59:03.715937 (+1220260us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:03.721572 (+ 5635us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:03.732238 (+ 10666us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.747311 (+ 15073us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:03.757636 (+ 10325us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2901,"prepare.queue_time_us":148805,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":10830,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":16779,"thread_start_us":1179625,"threads_started":2}
W20250114 08:59:03.897541 23330 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20250114 08:59:03.930658 21555 rpcz_store.cc:269] 0114 08:59:02.438148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.541891 (+103743us) service_pool.cc:225] Handling call
0114 08:59:02.938050 (+396159us) spinlock_profiling.cc:243] Waited 37.3 ms on lock 0x7b5400257c20. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f563bb1 00007fa63f5c4321 00007fa63f5c5b4e 00007fa63f5d1228 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02
0114 08:59:02.998542 (+ 60492us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b5400257c20. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f563020 00007fa63f5c433a 00007fa63f5c5b4e 00007fa63f5d1228 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02
0114 08:59:03.398507 (+399965us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b58000d9db0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5491c3 00007fa63f5acaba 00007fa63f5c43af 00007fa63f5c5b4e 00007fa63f5d1228 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c
0114 08:59:03.850689 (+452182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":660130944}
I20250114 08:59:03.879806 23051 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:03.949256 23353 log.cc:926] Time spent T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.057s user 0.002s sys 0.000s
W20250114 08:59:03.950256 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 184112us
I20250114 08:59:03.951079 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:03.951048 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:03.952131 21704 raft_consensus.cc:1270] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20250114 08:59:03.952597 23358 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:03.953370 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:03.953843 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 84960.1us
I20250114 08:59:03.954057 23309 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.955158 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1192) took 2822 ms. Trace:
I20250114 08:59:03.959139 23355 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250114 08:59:03.962385 23339 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.213s user 0.002s sys 0.000s
I20250114 08:59:03.962495 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:03.963994 23329 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:03.966576 23345 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:03.989526 21551 rpcz_store.cc:276] 0114 08:59:01.132868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.932585 (+1799717us) service_pool.cc:225] Handling call
0114 08:59:02.932884 (+ 299us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.954724 (+1021840us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.955039 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:04.003376 21628 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
W20250114 08:59:03.962587 23334 log.cc:926] Time spent T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.222s user 0.002s sys 0.003s
I20250114 08:59:03.922622 23074 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.012420 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
W20250114 08:59:04.018959 23363 log.cc:926] Time spent T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.165s user 0.000s sys 0.005s
I20250114 08:59:04.019488 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:04.019747 21697 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:59:04.020356 22270 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.019965 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1422) took 1105 ms. Trace:
I20250114 08:59:04.021296 21703 rpcz_store.cc:276] 0114 08:59:02.914747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.941011 (+ 26264us) service_pool.cc:225] Handling call
0114 08:59:03.012245 (+ 71234us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:03.012678 (+ 433us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:03.012718 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:03.074963 (+ 62245us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b2c00042708. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63dfae741 00007fa63dfbdd72 00007fa63f609141 00007fa63f5cd58a 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782
0114 08:59:03.075349 (+ 386us) log.cc:844] Serialized 26 byte log entry
0114 08:59:03.810757 (+735408us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:03.810871 (+ 114us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.861868 (+ 50997us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:04.019589 (+157721us) raft_consensus.cc:1680] finished
0114 08:59:04.019608 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.019893 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2038,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":140,"spinlock_wait_cycles":8635136,"thread_start_us":697890,"threads_started":1,"wal-append.queue_time_us":815690}
I20250114 08:59:04.022763 23329 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.022478 21623 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:04.023072 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:04.024220 21627 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:04.024437 21623 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
W20250114 08:59:04.026971 21629 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.532s user 0.039s sys 0.097s
W20250114 08:59:04.027344 21629 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.533s user 0.039s sys 0.097s
I20250114 08:59:04.030308 23355 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:04.036561 22270 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:04.039322 23074 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:04.040753 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 236505us
I20250114 08:59:04.043706 23309 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.045789 23257 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.041326 23074 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.029457 23051 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.061885 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1227) took 1666 ms. Trace:
I20250114 08:59:04.062803 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:04.064003 23309 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:04.064452 23309 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:04.064846 23309 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.066469 23355 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:04.067165 23355 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.069434 23355 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:04.073704 21668 leader_election.cc:336] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.779s (SENT)
I20250114 08:59:04.074368 21668 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.062562 21548 rpcz_store.cc:276] 0114 08:59:02.395309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.533015 (+1137706us) service_pool.cc:225] Handling call
0114 08:59:03.668129 (+135114us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:03.792247 (+124118us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:03.792289 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:03.809450 (+ 17161us) log.cc:844] Serialized 26 byte log entry
0114 08:59:03.927557 (+118107us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:03.927612 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.927703 (+ 91us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:04.061575 (+133872us) raft_consensus.cc:1680] finished
0114 08:59:04.061606 (+ 31us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.061804 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39501,"raft.queue_time_us":59049,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":5088,"thread_start_us":24059,"threads_started":1,"wal-append.queue_time_us":146654}
I20250114 08:59:04.075551 23232 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:03.863301 23211 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.078465 23211 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.078548 23307 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.084980 22925 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:04.085757 22925 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.086802 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.087610 21551 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:04.087895 22925 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.089206 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.090876 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.091881 21554 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:04.093353 23329 consensus_queue.cc:1046] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:59:04.093972 21623 raft_consensus.cc:1235] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 3. Current term is 4. Ops: []
W20250114 08:59:04.094360 23342 log.cc:926] Time spent T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20250114 08:59:04.094704 23329 raft_consensus.cc:3049] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:04.095161 23329 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:04.095643 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1160) took 1746 ms. Trace:
I20250114 08:59:04.087625 23358 raft_consensus.cc:695] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.098920 23307 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.098361 23358 consensus_queue.cc:237] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:03.874595 21547 rpcz_store.cc:276] 0114 08:59:02.408250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.571000 (+1162750us) service_pool.cc:225] Handling call
0114 08:59:03.597007 (+ 26007us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.757042 (+160035us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:03.759904 (+ 2862us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:03.777623 (+ 17719us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:03.780040 (+ 2417us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.788879 (+ 8839us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:03.796302 (+ 7423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:04.086941 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.096035 21630 rpcz_store.cc:276] 0114 08:59:02.349720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.485728 (+136008us) service_pool.cc:225] Handling call
0114 08:59:02.501804 (+ 16076us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.518328 (+ 16524us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.518378 (+ 50us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:02.537867 (+ 19489us) log.cc:844] Serialized 26 byte log entry
0114 08:59:03.783078 (+1245211us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:03.783157 (+ 79us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.797367 (+ 14210us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:04.095379 (+298012us) raft_consensus.cc:1680] finished
0114 08:59:04.095397 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.095560 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17775,"raft.queue_time_us":33031,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":13542,"thread_start_us":1129993,"threads_started":1,"wal-append.queue_time_us":1392743}
W20250114 08:59:04.099061 22881 consensus_meta.cc:287] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.716s user 0.012s sys 0.002s
I20250114 08:59:04.103286 22881 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:04.096339 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 79099us
W20250114 08:59:04.064227 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 48357.3us
I20250114 08:59:04.042974 21596 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:04.103966 21668 leader_election.cc:336] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.085s (SENT)
I20250114 08:59:04.114392 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1163) took 1761 ms. Trace:
I20250114 08:59:04.114508 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:04.114905 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:04.115567 23232 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.115911 21630 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.096199 23329 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:04.084255 23298 consensus_meta.cc:287] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.693s user 0.012s sys 0.000s
I20250114 08:59:04.139308 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:04.145480 23329 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.142365 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.158290 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:04.114934 21622 rpcz_store.cc:276] 0114 08:59:02.352814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.535422 (+182608us) service_pool.cc:225] Handling call
0114 08:59:02.535771 (+ 349us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.538616 (+ 2845us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.538663 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:02.539237 (+ 574us) log.cc:844] Serialized 26 byte log entry
0114 08:59:02.902314 (+363077us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b48000ce358. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa644491581 00007fa644488677 00007fa64448850a 00007fa64457c9bf 00007fa64457c687 00007fa644590b8c 00007fa644590b38 00007fa644590ab8 00007fa644590a73 00007fa64458fd1c 00007fa6434fcd16
0114 08:59:03.733251 (+830937us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:03.742014 (+ 8763us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.746631 (+ 4617us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:04.103907 (+357276us) raft_consensus.cc:1680] finished
0114 08:59:04.103935 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.114317 (+ 10382us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27225,"raft.queue_time_us":2458,"raft.run_cpu_time_us":3394,"raft.run_wall_time_us":362330,"spinlock_wait_cycles":11938560,"thread_start_us":1014089,"threads_started":1,"wal-append.queue_time_us":1546540}
I20250114 08:59:04.161393 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.158252 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.144915 22881 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.169034 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.139853 23183 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:04.184953 21704 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.185886 21548 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:04.137502 23298 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.190146 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.182296 21520 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.194242 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.195327 21549 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.195312 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.199239 21627 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:04.203234 21555 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 5
W20250114 08:59:04.179679 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 280126us
I20250114 08:59:04.206228 21697 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.211100 23309 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.211988 23346 consensus_queue.cc:1046] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
I20250114 08:59:04.215113 21594 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:04.216452 23047 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.223666 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:04.228941 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 186188us
W20250114 08:59:04.240912 21669 leader_election.cc:336] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.664s (SENT)
I20250114 08:59:04.198669 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.242770 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.246806 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.249971 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1221) took 1856 ms. Trace:
I20250114 08:59:04.250526 21550 rpcz_store.cc:276] 0114 08:59:02.393593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.491403 (+1097810us) service_pool.cc:225] Handling call
0114 08:59:03.549334 (+ 57931us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.549897 (+ 563us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 08:59:03.717724 (+167827us) log.cc:844] Serialized 14 byte log entry
0114 08:59:04.240478 (+522754us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:04.240552 (+ 74us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:04.249608 (+ 9056us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:04.249735 (+ 127us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.249897 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":433661,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":3377,"raft.queue_time_us":5872,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":3981,"thread_start_us":640312,"threads_started":2}
I20250114 08:59:04.251924 23300 consensus_queue.cc:1046] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:04.249974 23047 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.244184 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.257674 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.258525 21551 raft_consensus.cc:2388] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:04.258448 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.266235 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.261580 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.278225 21518 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:04.256611 23211 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.256879 21547 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.281595 22861 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:59:04.281597 23349 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250114 08:59:04.284796 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 130806us
I20250114 08:59:04.287645 21548 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:04.290153 23367 consensus_queue.cc:1046] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250114 08:59:04.261096 23225 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
W20250114 08:59:04.292865 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 50854.1us
W20250114 08:59:04.297741 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 82791.9us
I20250114 08:59:04.295008 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.302682 21702 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:04.291385 22269 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250114 08:59:04.323417 21699 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.324772 21701 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.317493 22925 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.331182 22885 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.322883 22843 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
I20250114 08:59:04.331849 22885 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.335522 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:04.341374 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 357669us
I20250114 08:59:04.328297 23298 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.327613 23349 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250114 08:59:04.344381 21703 raft_consensus.cc:2405] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:04.347764 22885 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.326071 23219 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.334002 22925 raft_consensus.cc:695] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.352270 23047 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:04.352437 23373 log.cc:926] Time spent T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20250114 08:59:04.351751 21596 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:04.345584 23309 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.354919 22861 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.003s
I20250114 08:59:04.365243 22925 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.368676 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.369494 21700 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.370604 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1267) took 2806 ms. Trace:
I20250114 08:59:04.377208 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1232) took 1940 ms. Trace:
I20250114 08:59:04.380753 21555 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:04.379684 21704 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:04.377746 21553 rpcz_store.cc:276] 0114 08:59:02.437127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.583337 (+1146210us) service_pool.cc:225] Handling call
0114 08:59:03.673423 (+ 90086us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:03.802848 (+129425us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:03.802894 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:03.803540 (+ 646us) log.cc:844] Serialized 26 byte log entry
0114 08:59:04.225410 (+421870us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:04.226045 (+ 635us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:04.226116 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:04.376982 (+150866us) raft_consensus.cc:1680] finished
0114 08:59:04.377002 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.377145 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67016,"raft.queue_time_us":68619,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"thread_start_us":421375,"threads_started":1,"wal-append.queue_time_us":528867}
I20250114 08:59:04.371029 21625 rpcz_store.cc:276] 0114 08:59:01.564111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.185899 (+621788us) service_pool.cc:225] Handling call
0114 08:59:02.408753 (+222854us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:02.427108 (+ 18355us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:02.427151 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:02.427611 (+ 460us) log.cc:844] Serialized 26 byte log entry
0114 08:59:02.496439 (+ 68828us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:03.680243 (+1183804us) log.cc:844] Serialized 14 byte log entry
0114 08:59:03.774978 (+ 94735us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:03.789322 (+ 14344us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:03.798067 (+ 8745us) raft_consensus.cc:1680] finished
0114 08:59:03.798087 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.370367 (+572280us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540022e050. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:04.370504 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47330,"prepare.queue_time_us":1466546,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":42750,"raft.run_cpu_time_us":506,"raft.run_wall_time_us":57635,"spinlock_wait_cycles":511479552,"thread_start_us":1157923,"threads_started":2,"wal-append.queue_time_us":140255,"wal-append.run_cpu_time_us":1900,"wal-append.run_wall_time_us":1118335}
I20250114 08:59:04.392700 21701 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:04.395485 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 72811.8us
I20250114 08:59:04.402227 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.405094 23325 raft_consensus.cc:3049] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:04.405731 23325 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:04.402235 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:04.402945 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.411172 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1215) took 2059 ms. Trace:
I20250114 08:59:04.425398 23208 raft_consensus.cc:3049] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:04.429773 23256 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:04.479662 23208 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:04.434769 23257 raft_consensus.cc:3049] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:04.480365 23257 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.480942 23208 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.481756 23257 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.482784 23208 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.483510 23257 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:04.435832 23337 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.882s user 0.003s sys 0.000s
W20250114 08:59:04.437554 23356 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.479s user 0.004s sys 0.000s
I20250114 08:59:04.470459 21552 rpcz_store.cc:276] 0114 08:59:02.351293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.249901 (+898608us) service_pool.cc:225] Handling call
0114 08:59:03.250916 (+ 1015us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:03.876703 (+625787us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:04.410815 (+534112us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b54002805d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:04.411088 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1063006592}
I20250114 08:59:04.486330 21698 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 4
W20250114 08:59:04.438441 23364 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.526s user 0.004s sys 0.000s
I20250114 08:59:04.445979 21520 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.449275 21549 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:04.391778 23219 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.456318 23183 consensus_queue.cc:1046] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250114 08:59:04.457368 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 130888us
I20250114 08:59:04.458082 23101 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:04.458583 21700 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:04.495512 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1150) took 1501 ms. Trace:
I20250114 08:59:04.495837 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:04.458802 22843 raft_consensus.cc:695] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.497555 23101 raft_consensus.cc:3049] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:04.498112 23101 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.498443 23219 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:04.498538 22843 consensus_queue.cc:237] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.500032 23047 raft_consensus.cc:3049] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:04.500499 23047 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.501452 23047 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:04.460525 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 120895us
I20250114 08:59:04.461988 23298 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:04.462929 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:04.463508 21630 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:04.464113 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.407076 23325 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.478842 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:04.479851 23256 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:04.509523 21703 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.510736 23051 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.511322 23051 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.512735 23052 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:04.513598 23051 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.513338 23052 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.496284 23350 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:04.502897 23047 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.515408 23052 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.515640 23298 raft_consensus.cc:695] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.516762 23298 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.499310 23101 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.497263 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:04.459121 23211 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:04.502408 21594 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:04.495599 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 49909.7us
I20250114 08:59:04.496086 21696 rpcz_store.cc:276] 0114 08:59:02.993630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:03.126475 (+132845us) service_pool.cc:225] Handling call
0114 08:59:03.127257 (+ 782us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:03.127691 (+ 434us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:03.127723 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:03.129059 (+ 1336us) log.cc:844] Serialized 508 byte log entry
0114 08:59:03.749790 (+620731us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:04.244215 (+494425us) log.cc:844] Serialized 14 byte log entry
0114 08:59:04.262451 (+ 18236us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:04.280728 (+ 18277us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:04.437057 (+156329us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b2400245c48. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c4eb9d4 00007fa63c4e9adb 00007fa63c578289 00007fa63c577f15 00007fa63c57755d 00007fa63f233d3e 00007fa63f24b4cc 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1
0114 08:59:04.487776 (+ 50719us) raft_consensus.cc:1680] finished
0114 08:59:04.495174 (+ 7398us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.495422 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27005,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":2073,"spinlock_wait_cycles":112760192,"thread_start_us":620203,"threads_started":1,"wal-append.queue_time_us":828322}
I20250114 08:59:04.506320 21699 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:04.433550 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1158) took 2117 ms. Trace:
I20250114 08:59:04.524127 21701 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250114 08:59:04.529129 22240 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.528779 21624 rpcz_store.cc:276] 0114 08:59:02.315948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.475687 (+159739us) service_pool.cc:225] Handling call
0114 08:59:04.433250 (+1957563us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54002918d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:04.433464 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2466884352}
I20250114 08:59:04.526010 21594 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:04.533809 21697 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:04.535416 21697 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:04.538815 21699 raft_consensus.cc:1235] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:04.522256 21521 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:04.532006 23183 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:04.548202 23219 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20250114 08:59:04.554108 23309 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.559212 23309 raft_consensus.cc:695] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.560567 23211 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:04.560405 23309 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:04.563841 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 49456.3us
I20250114 08:59:04.564780 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.542233 23345 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.569285 23345 raft_consensus.cc:695] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:04.570683 23345 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.552744 23183 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:04.573040 23183 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.568440 21520 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:04.528329 21547 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:04.586485 21703 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:04.591481 21594 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:04.585484 21521 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:04.578012 23101 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:04.602728 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 56272.2us
W20250114 08:59:04.604051 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 49095.8us
I20250114 08:59:04.580405 23232 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:04.607486 21521 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:04.593457 23307 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:59:04.617206 21704 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.617501 21698 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:04.629307 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:04.613394 22881 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 08:59:04.645828 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 77539.9us
I20250114 08:59:04.625941 23349 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:04.628721 23083 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:04.638186 22881 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:04.641216 23350 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.644942 23074 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:04.621109 23219 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
W20250114 08:59:04.665779 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 58390.2us
I20250114 08:59:04.671032 21521 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:04.675501 21594 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.664078 22881 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.645473 23256 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.628755 23378 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20250114 08:59:04.690634 21521 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:04.668030 23074 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:04.681669 23349 raft_consensus.cc:695] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.661127 23232 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 08:59:04.696363 23349 consensus_queue.cc:237] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.700402 22925 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.677611 23350 raft_consensus.cc:695] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.703402 21669 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.679028 22896 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20250114 08:59:04.704957 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.705917 21669 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.706396 22861 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:04.706758 23165 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:04.706874 22861 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:04.707080 21669 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.708045 23165 raft_consensus.cc:2758] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250114 08:59:04.709015 23165 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:04.709437 23165 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:04.709849 23165 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.710266 23256 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.714469 23325 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:04.702811 23350 consensus_queue.cc:237] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.721318 23165 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.707909 22861 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.708099 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.721268 22885 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:04.733749 22861 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.734066 22885 raft_consensus.cc:695] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.735108 22885 consensus_queue.cc:237] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.727090 21704 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:04.739807 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 114) took 2663 ms. Trace:
I20250114 08:59:04.692543 23211 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:04.694343 23074 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:04.754880 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 59918.9us
I20250114 08:59:04.733832 22925 raft_consensus.cc:695] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:04.765300 22925 consensus_queue.cc:237] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.740680 21629 rpcz_store.cc:276] 0114 08:59:02.076193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:02.484801 (+408608us) service_pool.cc:225] Handling call
0114 08:59:02.485122 (+ 321us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:02.485309 (+ 187us) raft_consensus.cc:1529] Early marking committed up to index 7
0114 08:59:04.027771 (+1542462us) log.cc:844] Serialized 14 byte log entry
0114 08:59:04.330278 (+302507us) log.cc:844] Serialized 14 byte log entry
0114 08:59:04.662661 (+332383us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:04.665507 (+ 2846us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:04.668893 (+ 3386us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:04.680848 (+ 11955us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:04.739717 (+ 58869us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":250620,"prepare.queue_time_us":1481520,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":136124,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":3432,"thread_start_us":2019488,"threads_started":3,"wal-append.queue_time_us":332831}
I20250114 08:59:04.753268 23211 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:04.754158 23378 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:59:04.780967 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 65783.3us
I20250114 08:59:04.750456 23183 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.782819 21594 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.775115 23165 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:04.778147 23211 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.792564 21594 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.805966 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.807435 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.808076 21554 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:04.809897 21668 leader_election.cc:336] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.542s (SENT)
I20250114 08:59:04.810364 21668 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.813045 23074 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.815160 23074 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:04.816105 23383 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
W20250114 08:59:04.817571 23379 log.cc:926] Time spent T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250114 08:59:04.802834 23367 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:04.821141 23183 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.821988 23367 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.821095 21554 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:04.825495 23358 raft_consensus.cc:988] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:59:04.826058 23358 raft_consensus.cc:1076] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Signalling peer d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:59:04.826758 23378 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:04.830840 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.827912 22896 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:04.837209 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.837994 21547 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:04.802013 23257 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.838804 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.839463 23257 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.839665 21553 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:04.840265 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:04.842136 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.830230 23047 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
W20250114 08:59:04.846491 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 138500us
I20250114 08:59:04.848956 21594 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.824294 23367 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:04.866187 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 75886us
I20250114 08:59:04.867040 21669 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.868259 23083 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.868886 23083 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.869012 23257 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.870185 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.871022 23083 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.853876 22896 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:04.873234 21624 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250114 08:59:04.872895 22896 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.873983 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:04.875733 22896 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:04.879228 23380 log.cc:926] Time spent T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.185s user 0.000s sys 0.002s
I20250114 08:59:04.882411 23052 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.882791 23300 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:04.882988 23052 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.883565 23074 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:04.883390 23300 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.870847 21547 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:04.885156 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.884093 22843 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:04.893817 23052 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:04.893707 22843 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:04.895848 23047 raft_consensus.cc:695] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.896401 21629 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.896672 21548 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:04.897004 23047 consensus_queue.cc:237] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:04.900858 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 51994.8us
I20250114 08:59:04.901995 23300 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:04.902084 21594 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.902626 23300 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:04.903115 23300 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:04.903059 21594 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:04.884222 23074 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.891251 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:04.912850 21552 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:04.895939 22843 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:04.911155 21629 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:04.936295 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:04.936717 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:04.921948 22896 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:04.939810 22896 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:04.940253 22896 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:04.941918 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.919536 23051 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:04.950214 21548 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:04.950258 21520 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:04.921602 23183 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:04.928185 23403 log.cc:926] Time spent T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20250114 08:59:04.922242 23083 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:04.951869 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:04.995574 21704 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250114 08:59:04.904052 23300 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:04.949898 23051 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:05.009055 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 67306.1us
I20250114 08:59:04.991106 23165 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:05.002858 23083 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:05.016187 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:05.023103 23391 log.cc:926] Time spent T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20250114 08:59:05.020057 23350 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20250114 08:59:05.025349 23378 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:05.025810 23378 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:05.026211 23378 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:05.029143 23397 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
W20250114 08:59:05.030874 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 117289us
I20250114 08:59:05.032178 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:05.032405 21547 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:05.012620 23399 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:05.023036 21548 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:05.011523 23051 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.022159 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.002955 23183 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.018927 23083 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.034588 22269 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250114 08:59:05.036518 22881 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.060204 23392 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:05.065699 23392 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:05.068027 23392 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.069711 21669 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.070432 23309 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:05.070976 23149 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:05.071439 23149 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:05.071768 23149 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.069777 23232 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 08:59:05.082630 23219 consensus_queue.cc:1046] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:05.083657 23149 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.084225 23074 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.057762 23165 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:05.043359 23399 raft_consensus.cc:2758] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20250114 08:59:05.087291 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 69542.3us
W20250114 08:59:05.133338 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 101272us
W20250114 08:59:05.148406 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 235139us
I20250114 08:59:05.152318 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:05.154662 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 83400us
I20250114 08:59:05.092667 23165 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:05.149987 21594 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.169596 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.167441 21704 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:05.177732 21521 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:05.161691 23165 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.182407 22266 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:59:05.194859 23047 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:05.181205 23074 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:05.197276 23074 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:05.198482 23183 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.198202 23074 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.181349 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:05.200932 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 76836.9us
W20250114 08:59:05.182403 23407 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250114 08:59:05.202818 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.203651 23345 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:05.204063 23345 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:05.204433 23345 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.182664 23399 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:05.204914 23399 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:05.183619 23408 log.cc:926] Time spent T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20250114 08:59:05.184646 21520 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:05.185236 22881 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:05.188583 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 69429.2us
I20250114 08:59:05.190322 22896 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.174165 23101 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:05.224938 22896 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:05.226163 21553 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250114 08:59:05.227274 23387 log.cc:926] Time spent T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.512s user 0.008s sys 0.004s
I20250114 08:59:05.227996 23411 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:05.227299 21696 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:05.228932 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.205358 23399 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:59:05.251987 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 74379.2us
I20250114 08:59:05.258255 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.258257 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.259078 21554 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:05.222637 22885 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:05.284103 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.284108 21550 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:05.253085 23052 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:05.293350 21698 raft_consensus.cc:1235] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:05.290849 21551 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250114 08:59:05.294183 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 109722us
I20250114 08:59:05.292505 21700 raft_consensus.cc:1270] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:05.256152 23409 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:05.256616 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.304536 23052 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.228559 23411 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.310591 23052 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.316090 21552 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:05.317139 21695 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250114 08:59:05.304867 23211 consensus_meta.cc:287] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.505s user 0.009s sys 0.001s
I20250114 08:59:05.319365 23378 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.303387 23409 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:05.324159 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 147709us
I20250114 08:59:05.305581 23367 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250114 08:59:05.312116 23411 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.349496 23165 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.259285 23083 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.375169 23349 consensus_queue.cc:1046] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.009s
I20250114 08:59:05.380298 23419 consensus_queue.cc:1046] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250114 08:59:05.384698 23419 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:05.385109 23419 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:05.386232 23378 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:05.385933 23419 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.387295 23419 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:05.363188 23307 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250114 08:59:05.379123 21698 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:05.383100 21704 raft_consensus.cc:1270] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20250114 08:59:05.402782 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 100976us
I20250114 08:59:05.408953 21669 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.351518 23211 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.490375 23400 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:05.498836 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:05.415124 23101 raft_consensus.cc:695] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:05.378782 23165 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:05.518162 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.415839 23349 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:05.526877 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:05.402150 23051 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.527889 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:05.530058 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.530928 21552 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:05.412951 23415 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.119s
I20250114 08:59:05.538689 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:05.542448 21629 raft_consensus.cc:2388] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:05.542863 21630 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:05.498811 23424 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:05.538962 21594 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.422590 23345 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:05.549793 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.538266 22885 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:05.553493 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.558086 21702 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:05.532076 23349 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.532910 23300 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:05.562350 23300 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:05.562610 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.562863 23300 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:05.563407 21697 raft_consensus.cc:2388] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:05.564272 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:05.564956 21697 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:05.565076 22885 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.549877 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:05.559846 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.544319 22843 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:05.545553 23232 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:05.567441 21696 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:05.568501 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 210441us
I20250114 08:59:05.569291 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:05.569990 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 147262us
I20250114 08:59:05.574453 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.575434 21624 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 08:59:05.576915 21700 raft_consensus.cc:1270] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:05.578845 21699 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:05.579660 23211 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.580229 23422 raft_consensus.cc:3049] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:05.580636 23422 raft_consensus.cc:738] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:05.559228 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.581732 21547 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:05.581616 23422 consensus_queue.cc:260] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.582707 23422 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.527880 23430 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250114 08:59:05.571638 23083 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.588063 23149 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:05.548663 23435 raft_consensus.cc:2758] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250114 08:59:05.588558 23232 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.567287 23424 raft_consensus.cc:695] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:05.563776 23300 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.580691 23367 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20250114 08:59:05.627151 23350 consensus_queue.cc:1046] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:05.627403 23435 consensus_queue.cc:1035] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:05.628245 23350 raft_consensus.cc:3049] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:05.629153 23350 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:05.590911 23345 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:05.579881 23399 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:05.592679 23051 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.593942 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:05.594278 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.608160 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.696244 21703 raft_consensus.cc:2405] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:05.610879 21702 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:05.657156 23409 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.593787 23431 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:05.559226 23421 consensus_queue.cc:1035] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.295s
I20250114 08:59:05.659788 23232 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:05.660806 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:05.602851 22843 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:05.664924 21668 leader_election.cc:336] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 3.058s (SENT)
I20250114 08:59:05.593839 23101 consensus_queue.cc:237] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.664155 21520 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:05.702977 21699 raft_consensus.cc:2388] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 3: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:05.699666 21668 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:05.667840 22885 consensus_queue.cc:1035] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:05.591375 23349 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:05.679208 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 121894us
I20250114 08:59:05.697422 21697 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
W20250114 08:59:05.723784 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 166504us
I20250114 08:59:05.724745 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.745651 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:05.746344 21699 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
W20250114 08:59:05.748950 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 124639us
I20250114 08:59:05.699061 23431 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:05.756439 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:05.757405 21697 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:05.745442 23385 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:05.765676 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 100788us
I20250114 08:59:05.768080 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.768169 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:05.763756 23083 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:05.776304 23083 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:05.776860 23083 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:05.771797 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:05.782917 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:05.749156 23438 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:59:05.785755 23345 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:05.786037 22843 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.786623 23424 consensus_queue.cc:237] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.782601 23350 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.756971 22267 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:05.795612 23438 consensus_queue.cc:1046] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:05.796640 21521 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:05.799914 23423 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.271s user 0.000s sys 0.006s
I20250114 08:59:05.750075 23232 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250114 08:59:05.857443 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 88255.2us
I20250114 08:59:05.860330 21594 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:05.761365 23431 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:05.939800 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 275751us
I20250114 08:59:05.942675 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:05.947716 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 89356.7us
I20250114 08:59:05.950671 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:05.951385 21547 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.952524 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:05.955077 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:05.955724 21554 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.956992 23399 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.940928 22843 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:05.957921 22843 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:05.958276 21547 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:05.959968 22843 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:05.960628 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.961537 21548 raft_consensus.cc:2405] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:05.965915 21554 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:05.902639 23438 raft_consensus.cc:3049] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:05.905519 23052 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:05.967176 23257 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:05.968094 21550 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250114 08:59:05.968935 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:05.993920 21546 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:05.970346 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:05.973865 21553 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:05.859774 22885 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:05.923736 23350 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:05.883188 23345 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:05.981274 23257 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:05.992929 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:05.999819 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.002786 21702 raft_consensus.cc:2388] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:05.975111 23443 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:05.971937 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 187650us
I20250114 08:59:05.980661 23438 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:05.994781 23052 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.021344 23211 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:06.037147 23438 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.038187 23052 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.037936 23211 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.031077 22241 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:06.038486 23438 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.013454 23257 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.042174 23441 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:06.042836 23441 raft_consensus.cc:695] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:06.000255 22885 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:06.045375 22885 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:06.045611 23329 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250114 08:59:06.032555 21629 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20250114 08:59:06.021392 23443 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.014844 23083 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.047158 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.047802 21629 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:06.048188 23083 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.014083 23345 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.049285 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.050426 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.051610 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:06.052191 21696 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.033948 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.034211 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.055732 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:06.056347 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.048188 23443 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.035187 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.057497 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.058133 21554 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:06.058198 21702 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:06.058740 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.053085 22270 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250114 08:59:06.059419 21627 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:06.059715 22270 pending_rounds.cc:77] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Aborting all ops after (but not including) 5
I20250114 08:59:06.060071 22270 pending_rounds.cc:99] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250114 08:59:06.034447 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.040210 23211 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:06.060753 21621 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.062932 23417 raft_consensus.cc:3049] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:06.063347 23417 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:06.063942 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.064180 23417 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.057112 23369 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:06.066486 23369 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.068567 23369 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.032873 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.070191 21623 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.065488 21594 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.052378 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:06.074213 21704 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.050539 23345 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.059085 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.081248 21702 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:06.095620 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.060353 22270 raft_consensus.cc:2883] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:05.867285 23385 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.005450 23442 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:05.994705 23411 consensus_queue.cc:1046] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20250114 08:59:06.101765 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.083567 23443 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.104139 21701 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:59:06.104502 21594 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.065546 23417 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:06.097692 21629 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:06.110713 21702 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:59:06.111876 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.114653 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:06.115232 21701 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.043882 23441 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.119874 21621 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:06.106509 23422 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:06.127415 21696 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:06.107972 23436 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:06.130164 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.134980 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.164302 21629 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:06.121399 23443 raft_consensus.cc:695] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:06.161998 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.146515 21704 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:06.168352 21627 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:59:06.168807 21702 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:06.169811 21521 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:06.171378 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.173452 21701 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:06.175021 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.175040 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.175738 21621 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:06.176353 21594 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.176812 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.177564 21621 raft_consensus.cc:2388] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:06.177671 23074 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:06.178180 23257 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.178498 22885 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.179541 21552 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:59:06.180899 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.180558 23436 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.181941 23422 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:06.182399 21668 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.929s (SENT) [suppressed 28 similar messages]
I20250114 08:59:06.183570 23436 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.184042 23422 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.188756 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:06.189399 21699 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.189494 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.184331 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.175696 21629 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.195725 21701 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:06.198853 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.199568 21696 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:06.208721 21521 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:06.163800 23052 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.213279 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.214476 21627 raft_consensus.cc:2388] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:06.215104 21704 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:06.180320 23257 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.219616 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.220317 21621 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.210414 23436 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:06.223616 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.195887 23358 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.208366 22843 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:06.228700 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.232836 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.236635 21623 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:06.263453 21551 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:06.238554 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.266943 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.222183 23422 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:06.270776 21550 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:06.286051 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.222488 23211 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:06.234349 23074 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:06.229152 23052 raft_consensus.cc:695] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:06.222230 23256 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.063s
I20250114 08:59:06.221882 23225 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20250114 08:59:06.234025 22843 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:06.303180 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:06.309615 21696 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20250114 08:59:06.312779 21696 pending_rounds.cc:77] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:59:06.313104 21696 pending_rounds.cc:99] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:06.225438 23436 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:06.312788 21554 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.308398 21699 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:06.313575 21696 raft_consensus.cc:2883] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:06.318233 21696 pending_rounds.cc:99] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:06.318559 21696 raft_consensus.cc:2883] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:06.284237 23358 raft_consensus.cc:695] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:06.292537 23422 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:06.319835 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.306960 21621 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:06.325268 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.268944 23400 consensus_queue.cc:1046] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20250114 08:59:06.326452 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.397423 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.327414 23422 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:06.343315 22885 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.288066 23257 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20250114 08:59:06.345273 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.345306 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.439422 21551 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:06.316987 22843 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:06.347244 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.353566 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:06.442663 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 89174.2us
I20250114 08:59:06.310456 23211 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.317257 23436 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:06.398145 21704 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:06.309305 23074 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:06.419196 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 236795us
I20250114 08:59:06.446777 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:06.447566 21701 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:06.447806 21668 leader_election.cc:336] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.699s (SENT)
I20250114 08:59:06.396857 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.445713 21696 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:06.450809 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.454237 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.455274 21621 raft_consensus.cc:2388] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:06.448949 21696 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:06.293648 23443 consensus_queue.cc:237] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.465881 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:06.470975 21704 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.468547 21669 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.338480 21629 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:06.473773 21699 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
W20250114 08:59:06.436969 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 265636us
I20250114 08:59:06.438540 23349 raft_consensus.cc:3049] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:06.475955 21701 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:06.478260 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.480552 21621 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:06.479055 21554 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:06.480844 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.486286 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:06.489926 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.491062 21629 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:06.481534 21551 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:06.503827 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.504731 21554 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:06.492383 21594 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.502904 21704 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
W20250114 08:59:06.596218 23438 consensus_meta.cc:287] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.557s user 0.012s sys 0.000s
W20250114 08:59:06.597157 23465 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20250114 08:59:06.506361 23052 consensus_queue.cc:237] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:06.604813 23457 log.cc:926] Time spent T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.260s user 0.001s sys 0.000s
W20250114 08:59:06.605083 23454 log.cc:926] Time spent T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20250114 08:59:06.605796 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.477053 21696 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.606120 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.606694 21696 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:06.606979 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.607565 21622 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:06.608369 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.609390 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.610376 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.610491 23417 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.611037 23417 raft_consensus.cc:695] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:06.611362 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.611907 21622 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:06.612690 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 120406us
W20250114 08:59:06.613049 23458 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.251s user 0.000s sys 0.000s
I20250114 08:59:06.613101 23444 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.613765 23444 raft_consensus.cc:695] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:06.613642 23256 consensus_queue.cc:1046] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:06.614259 23358 consensus_queue.cc:237] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:06.614718 23444 consensus_queue.cc:237] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.615304 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.614650 23256 raft_consensus.cc:3049] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:06.605907 23211 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.616094 23256 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:06.479841 23349 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:06.617292 23349 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.617451 23256 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:06.617975 23426 log.cc:926] Time spent T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 1.121s user 0.008s sys 0.007s
I20250114 08:59:06.618602 23256 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:06.612864 21629 raft_consensus.cc:2405] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:06.622427 21696 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:06.625196 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:06.625975 21699 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:06.469341 23424 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:06.661550 23385 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:06.470557 23425 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.192s
I20250114 08:59:06.631611 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.633769 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1265) took 1412 ms. Trace:
I20250114 08:59:06.633927 21695 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
W20250114 08:59:06.673118 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 204694us
I20250114 08:59:06.634421 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.642865 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.645527 23074 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.646282 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.651017 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:06.677484 21553 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:06.652233 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:06.655637 23431 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.680s user 0.002s sys 0.007s
I20250114 08:59:06.655745 21622 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:06.658833 23350 consensus_meta.cc:287] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.653s user 0.009s sys 0.000s
I20250114 08:59:06.668850 21554 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:06.691804 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.668758 21555 rpcz_store.cc:276] 0114 08:59:05.221660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:05.223810 (+ 2150us) service_pool.cc:225] Handling call
0114 08:59:05.224181 (+ 371us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:05.224382 (+ 201us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:05.224405 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:05.224871 (+ 466us) log.cc:844] Serialized 26 byte log entry
0114 08:59:05.289460 (+ 64589us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:05.289504 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:05.289563 (+ 59us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:06.312573 (+1023010us) spinlock_profiling.cc:243] Waited 53 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:06.619021 (+306448us) raft_consensus.cc:1680] finished
0114 08:59:06.619045 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:06.633685 (+ 14640us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121881472,"thread_start_us":64241,"threads_started":1,"wal-append.queue_time_us":137831}
I20250114 08:59:06.633100 21621 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:06.678421 21628 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.676844 21669 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.684903 21623 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
W20250114 08:59:06.696139 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 248360us
I20250114 08:59:06.717183 21621 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:06.720206 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:06.721487 21622 raft_consensus.cc:2405] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:06.724431 21629 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:06.724356 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.726388 21622 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:06.618602 23349 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:06.673099 21695 pending_rounds.cc:77] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:59:06.611842 23417 consensus_queue.cc:237] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:06.701326 23459 log.cc:926] Time spent T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20250114 08:59:06.694427 21546 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:59:06.736258 21622 raft_consensus.cc:1235] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:06.748338 21624 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:06.737413 21629 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:59:06.738907 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:06.734045 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:06.752861 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 149285us
I20250114 08:59:06.730293 21695 pending_rounds.cc:99] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:06.762735 21621 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250114 08:59:06.761672 21596 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:06.773970 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.777817 21554 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250114 08:59:06.768959 21695 raft_consensus.cc:2883] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:06.784505 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:06.716310 23431 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.775797 21622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:57318
I20250114 08:59:06.795207 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.796033 21630 raft_consensus.cc:1270] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:59:06.798348 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.785853 23424 raft_consensus.cc:695] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:06.792872 23298 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:06.830380 21622 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:06.838786 21622 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:06.831576 21555 raft_consensus.cc:2405] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:06.834228 21553 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:06.835049 23449 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:06.837257 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 160489us
I20250114 08:59:06.794884 23472 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:06.849244 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 64879us
I20250114 08:59:06.864658 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:06.851007 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.861863 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.830179 21695 pending_rounds.cc:99] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:06.868660 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:06.884438 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:06.893261 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:06.896171 21622 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:06.899813 21622 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.902400 21622 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:06.905287 21622 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:06.906401 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:06.906500 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 108272us
W20250114 08:59:06.907632 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 244043us
I20250114 08:59:06.910313 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:06.883443 23464 log.cc:926] Time spent T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250114 08:59:06.915323 23483 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:06.938455 23431 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:06.941457 23479 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:06.943174 21626 raft_consensus.cc:1270] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 08:59:06.883647 21695 raft_consensus.cc:2883] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:06.816309 22843 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:06.948262 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 73112.3us
I20250114 08:59:06.865922 23471 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:06.883941 23425 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.953811 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.954708 21555 raft_consensus.cc:2405] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:06.955313 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:06.956179 23482 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:06.949008 21622 raft_consensus.cc:1270] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:06.944089 23483 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.916059 23101 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:06.953020 21628 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:06.960125 21624 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:06.907179 21546 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:06.971994 21627 raft_consensus.cc:2388] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:06.976115 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:06.977401 21555 raft_consensus.cc:2405] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:06.950812 23411 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:06.986868 23047 raft_consensus.cc:3049] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:06.959734 23101 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:06.993855 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:06.906235 23449 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:06.869922 23472 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.006232 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.005427 23047 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:07.013998 23431 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250114 08:59:07.007057 23298 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.012888 23101 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:07.045127 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 177561us
W20250114 08:59:07.047046 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 84646us
I20250114 08:59:07.048296 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.052093 21549 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:07.059177 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.059973 21555 raft_consensus.cc:2371] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4. Re-sending same reply.
I20250114 08:59:07.059935 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:07.021538 23483 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:06.954768 23479 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.039983 23425 raft_consensus.cc:695] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:06.981364 23470 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.086s
I20250114 08:59:07.112440 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.064311 23385 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:06.957355 23424 consensus_queue.cc:237] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.047897 23444 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:07.124241 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.124404 21549 raft_consensus.cc:2371] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4. Re-sending same reply.
W20250114 08:59:07.119097 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 60075.9us
I20250114 08:59:07.047391 23482 raft_consensus.cc:695] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:07.126052 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 77832.2us
I20250114 08:59:07.132331 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.067555 23438 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.066231 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:07.148852 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 100118us
I20250114 08:59:07.152081 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.153235 21555 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:07.144477 23047 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.156843 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.160102 21520 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.188606 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.272998 21549 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:07.275360 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.149627 23400 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:07.149791 22925 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:07.131374 23422 consensus_meta.cc:287] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.685s user 0.016s sys 0.003s
W20250114 08:59:07.286904 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 137637us
W20250114 08:59:07.287107 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 314308us
W20250114 08:59:07.210309 23486 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
W20250114 08:59:07.203755 23488 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20250114 08:59:07.202939 23489 log.cc:926] Time spent T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250114 08:59:07.288504 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:07.289469 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:07.289779 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 140227us
I20250114 08:59:07.294903 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.291781 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.268543 23479 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:07.212563 23449 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:07.272318 22843 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.297960 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.154614 23444 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:07.154320 21546 raft_consensus.cc:2371] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7. Re-sending same reply.
I20250114 08:59:07.301869 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.286590 23358 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.157635 23494 consensus_queue.cc:1046] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.174s
I20250114 08:59:07.328392 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.201869 23480 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.105s
I20250114 08:59:07.296698 23047 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:07.297631 23441 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:07.333339 23484 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20250114 08:59:07.329648 21669 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:07.338385 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 46693.1us
I20250114 08:59:07.339459 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.302641 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:07.340124 22843 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:07.286906 23400 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:07.344036 23349 consensus_meta.cc:287] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.612s user 0.011s sys 0.000s
I20250114 08:59:07.239872 23482 consensus_queue.cc:237] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.344846 21554 raft_consensus.cc:2363] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4.
I20250114 08:59:07.345322 23400 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.345911 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.346765 21546 raft_consensus.cc:2363] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4.
I20250114 08:59:07.341787 23441 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.347635 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.324757 23449 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.286566 23422 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.324052 23431 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 08:59:07.352353 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.353174 21549 raft_consensus.cc:3049] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:59:07.353526 21549 raft_consensus.cc:738] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:07.302804 23417 consensus_queue.cc:1046] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20250114 08:59:07.345922 23477 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250114 08:59:07.355122 22267 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:07.359097 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.363353 22239 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:07.200935 23425 consensus_queue.cc:237] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.364920 23442 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:07.365314 23442 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:07.365702 23442 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:07.360345 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.375614 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.376987 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.378551 21702 raft_consensus.cc:1270] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:07.378829 22271 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:07.372314 21596 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.381300 21701 raft_consensus.cc:1270] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:59:07.382653 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:07.383273 21696 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:07.385147 21548 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:07.384982 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.381578 21549 consensus_queue.cc:260] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.355072 23501 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:07.397769 21700 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:07.398350 21596 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.396755 23480 consensus_queue.cc:1225] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:07.399052 21667 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:07.402704 21549 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:07.406739 21596 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.385773 23417 consensus_queue.cc:1225] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:07.410415 21596 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.405592 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.394165 23309 consensus_queue.cc:1035] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 08:59:07.412673 23497 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.412882 23502 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250114 08:59:07.413421 23256 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:59:07.414373 21547 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:07.415002 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1293) took 1199 ms. Trace:
I20250114 08:59:07.415304 21547 rpcz_store.cc:276] 0114 08:59:06.215239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.219399 (+ 4160us) service_pool.cc:225] Handling call
0114 08:59:07.414029 (+1194630us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b5800000f98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656d0e 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:07.414193 (+ 164us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:07.414760 (+ 567us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:07.414939 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65118592}
I20250114 08:59:07.394289 23421 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20250114 08:59:07.417726 23491 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:07.392741 23400 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:07.410099 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.421401 21696 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:07.420562 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.422422 21550 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:07.422768 21521 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:07.423729 22925 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:07.397351 23051 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.424942 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:07.430363 21696 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 6
W20250114 08:59:07.424494 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 94906.5us
W20250114 08:59:07.412685 23504 log.cc:926] Time spent T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250114 08:59:07.409440 22843 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.405356 23441 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.431349 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.421063 23474 raft_consensus.cc:3049] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:07.446632 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.450192 21555 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:07.422364 23480 raft_consensus.cc:3049] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:07.420388 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:07.441886 23436 consensus_meta.cc:287] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.972s user 0.014s sys 0.000s
I20250114 08:59:07.429456 23422 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.411203 23349 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20250114 08:59:07.462813 21596 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:07.463188 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:07.430119 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.465476 21520 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.466990 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.445986 23497 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.440827 22925 raft_consensus.cc:695] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:07.459082 23480 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:07.450659 23474 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:07.467514 21547 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 5
I20250114 08:59:07.473338 22843 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:07.473541 22270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 149) took 1559 ms. Trace:
I20250114 08:59:07.499624 22270 rpcz_store.cc:276] 0114 08:59:05.914587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:05.914764 (+ 177us) service_pool.cc:225] Handling call
0114 08:59:05.915196 (+ 432us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:06.122882 (+207686us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:07.473262 (+1350380us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400599f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:07.473457 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316175104}
I20250114 08:59:07.403721 23444 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.479486 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1301) took 1292 ms. Trace:
I20250114 08:59:07.468492 21549 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:07.483588 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.476121 21553 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:07.470000 23487 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.489770 21630 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:07.502157 21623 rpcz_store.cc:276] 0114 08:59:06.187176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.322779 (+135603us) service_pool.cc:225] Handling call
0114 08:59:07.479305 (+1156526us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b54001658d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:07.479421 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86762752}
W20250114 08:59:07.493395 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 94410.1us
I20250114 08:59:07.497336 21696 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:07.518574 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.503502 21551 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:07.498147 21520 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.504832 21594 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:07.506350 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.491492 21596 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.504225 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.519273 21696 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:07.520988 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.522540 21549 raft_consensus.cc:2388] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:07.457973 23425 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20250114 08:59:07.485883 23421 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.536412 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.537973 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.538540 21697 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 08:59:07.540220 23512 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:07.539907 22925 consensus_queue.cc:237] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.543183 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.544855 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.546381 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.547161 23442 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.542954 23482 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:07.409963 23501 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.525844 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.464854 23471 consensus_queue.cc:1046] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
W20250114 08:59:07.544950 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 46935.1us
I20250114 08:59:07.525327 21521 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:07.524614 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:07.547659 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:07.549729 21596 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:07.468182 23436 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.632784 21549 raft_consensus.cc:2363] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 3. Current term is 4.
W20250114 08:59:07.635434 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 110188us
I20250114 08:59:07.570531 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:07.639716 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 113969us
W20250114 08:59:07.549077 23514 log.cc:926] Time spent T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250114 08:59:07.649633 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.552175 23497 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.662492 21623 raft_consensus.cc:1270] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:07.551756 23431 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader election won for term 2
W20250114 08:59:07.668262 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 118632us
I20250114 08:59:07.669349 21596 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:07.524061 22843 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:07.613287 23510 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:07.548206 23422 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:07.650571 23487 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:07.612277 23421 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:07.585793 23491 raft_consensus.cc:695] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:07.678236 21699 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:07.648675 23490 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:07.683781 21623 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:07.670923 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:07.696028 21550 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:07.697999 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:07.702908 21550 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250114 08:59:07.702044 21553 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:07.702647 21594 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:07.549964 23047 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
I20250114 08:59:07.666486 23471 consensus_queue.cc:1225] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250114 08:59:07.705838 21622 raft_consensus.cc:1212] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Deduplicated request from leader. Original: 1.1->[4.2-4.2] Dedup: 4.2->[]
I20250114 08:59:07.710484 21622 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:07.595590 23449 consensus_queue.cc:1035] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20250114 08:59:07.680161 23051 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250114 08:59:07.699514 23101 consensus_meta.cc:287] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.636s user 0.012s sys 0.000s
I20250114 08:59:07.679920 23256 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.713099 21596 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:07.696985 23431 raft_consensus.cc:695] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:07.684374 23438 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.697304 21630 raft_consensus.cc:2949] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:07.707456 23487 raft_consensus.cc:695] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:07.724824 23083 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:07.716647 23349 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:07.716706 23051 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:07.742863 23083 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:07.746026 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 389630us
I20250114 08:59:07.750321 21550 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:07.712333 23424 consensus_queue.cc:1035] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20250114 08:59:07.733997 23225 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:06.396836 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:07.760125 23526 consensus_queue.cc:1046] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20250114 08:59:07.759469 23500 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
W20250114 08:59:07.761466 23511 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.233s user 0.001s sys 0.000s
I20250114 08:59:07.760896 23479 raft_consensus.cc:3049] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:07.753774 23083 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:07.751658 23051 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:07.771709 23225 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:07.803774 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:07.731891 23101 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.753814 23513 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:07.747354 23471 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20250114 08:59:07.823426 21629 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:07.745378 23498 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.247s
I20250114 08:59:07.834349 23501 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:07.834456 23522 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.2, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20250114 08:59:07.812956 23500 raft_consensus.cc:695] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:07.835067 23501 consensus_queue.cc:1046] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:07.835390 23522 consensus_queue.cc:1225] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:07.804014 23479 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:07.846904 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:07.821812 23225 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:07.755211 23490 consensus_queue.cc:1046] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.072s
I20250114 08:59:07.540788 23512 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.861703 23512 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:07.862447 21548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1292) took 1647 ms (client timeout 2182 ms). Trace:
W20250114 08:59:07.862872 21548 rpcz_store.cc:269] 0114 08:59:06.214954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.215479 (+ 525us) service_pool.cc:225] Handling call
0114 08:59:07.862252 (+1646773us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540028f5d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:07.862388 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2644206336}
I20250114 08:59:07.863334 23083 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.699791 23421 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.866174 23421 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.762205 23487 consensus_queue.cc:237] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.826505 23431 consensus_queue.cc:237] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.867626 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1248) took 1050 ms. Trace:
I20250114 08:59:07.867998 21554 rpcz_store.cc:276] 0114 08:59:06.816719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.878964 (+ 62245us) service_pool.cc:225] Handling call
0114 08:59:07.867446 (+988482us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b540031cc54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:07.867570 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":997397248}
I20250114 08:59:07.859737 23442 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.851775 23436 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.879880 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1299) took 1395 ms. Trace:
I20250114 08:59:07.831499 21453 catalog_manager.cc:5526] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 3, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:07.880317 21552 rpcz_store.cc:276] 0114 08:59:06.484514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.487348 (+ 2834us) service_pool.cc:225] Handling call
0114 08:59:06.487765 (+ 417us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:06.506012 (+ 18247us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:06.506048 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:06.506503 (+ 455us) log.cc:844] Serialized 26 byte log entry
0114 08:59:07.333076 (+826573us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b1800056e00. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:59:07.400811 (+ 67735us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:07.400846 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:07.400909 (+ 63us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:07.879576 (+478667us) raft_consensus.cc:1680] finished
0114 08:59:07.879596 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:07.879817 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2523,"raft.queue_time_us":18675,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":44338560,"thread_start_us":67564,"threads_started":1,"wal-append.queue_time_us":862261}
I20250114 08:59:07.704676 23510 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.501564 23474 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.804925 22843 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.629417 23480 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.771340 23491 consensus_queue.cc:237] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.880823 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:07.906970 21629 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
W20250114 08:59:07.922144 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 170786us
I20250114 08:59:07.923936 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:07.933409 21697 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:07.932672 23442 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:07.938670 23442 raft_consensus.cc:695] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:07.835999 23500 consensus_queue.cc:237] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.939647 23442 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:07.948648 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:07.889087 23479 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:07.944921 23083 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.918992 23533 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250114 08:59:07.979861 21547 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:07.956060 23101 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.992417 23421 consensus_queue.cc:1046] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:07.994053 23421 consensus_queue.cc:1225] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250114 08:59:07.994598 23421 raft_consensus.cc:3049] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:07.994694 23510 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:07.995004 23421 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:08.010489 23476 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:08.011160 23476 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:08.011862 23518 raft_consensus.cc:3049] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:07.983541 23479 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:07.995893 23421 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:08.014443 23449 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250114 08:59:08.009824 23329 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:08.022030 23329 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:08.024143 23329 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:08.024508 23472 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:08.012240 23518 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:08.024883 23494 raft_consensus.cc:3049] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:08.026697 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:08.029080 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:08.029471 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:08.014201 23513 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:08.031873 23513 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:08.029909 22269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 151) took 1356 ms. Trace:
I20250114 08:59:08.033084 22269 rpcz_store.cc:276] 0114 08:59:06.673796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.723245 (+ 49449us) service_pool.cc:225] Handling call
0114 08:59:08.029219 (+1305974us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656dac 00007fa63c656d16 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:08.029352 (+ 133us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:08.029533 (+ 181us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:08.029564 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:08.029610 (+ 46us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:08.029642 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:08.029719 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:08.029849 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4128640}
I20250114 08:59:08.014170 23476 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:08.031026 23421 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:08.031356 23494 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:08.012208 22241 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:08.035579 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:08.053045 23490 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:08.039899 23051 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:08.070326 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1201) took 2055 ms. Trace:
I20250114 08:59:08.060444 21547 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:08.084910 21625 rpcz_store.cc:276] 0114 08:59:06.014729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.015014 (+ 285us) service_pool.cc:225] Handling call
0114 08:59:06.044899 (+ 29885us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:06.069731 (+ 24832us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:06.069774 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:06.070289 (+ 515us) log.cc:844] Serialized 52 byte log entry
0114 08:59:06.231292 (+161003us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:06.475408 (+244116us) log.cc:844] Serialized 14 byte log entry
0114 08:59:06.475677 (+ 269us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:06.475770 (+ 93us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:06.793935 (+318165us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b1800211600. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:08.067817 (+1273882us) spinlock_profiling.cc:243] Waited 8.81 ms on lock 0x7b2400158be8. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c4eb9d4 00007fa63c4e9adb 00007fa63c578289 00007fa63c577f15 00007fa63c577470 00007fa63f277f88 00007fa63f2786e8 00007fa63f24ff98 00007fa63f24b3aa 00007fa63f24a7f6
0114 08:59:08.070117 (+ 2300us) raft_consensus.cc:1680] finished
0114 08:59:08.070136 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:08.070269 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65636,"prepare.queue_time_us":104644,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":42,"raft.queue_time_us":26876,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":11192,"spinlock_wait_cycles":44720128,"thread_start_us":334898,"threads_started":2,"wal-append.queue_time_us":258906}
I20250114 08:59:08.126324 21629 raft_consensus.cc:1235] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:08.129616 23051 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:08.131981 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
W20250114 08:59:08.068061 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 55958.9us
I20250114 08:59:08.063635 23518 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:08.161449 23518 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:08.069818 23490 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:08.086486 23225 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:08.157007 23494 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:08.863953 21548 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.981s user 0.011s sys 0.039s
W20250114 08:59:09.029513 21554 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.014s sys 0.007s
W20250114 08:59:09.051298 21548 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.169s user 0.011s sys 0.052s
W20250114 08:59:09.051512 21554 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.119s user 0.014s sys 0.007s
W20250114 08:59:08.603929 21549 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.014s sys 0.018s
W20250114 08:59:09.052408 21549 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.228s user 0.020s sys 0.021s
W20250114 08:59:08.484570 21553 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.014s sys 0.035s
W20250114 08:59:08.909834 21594 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.177s user 0.016s sys 0.020s
W20250114 08:59:09.029340 21550 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.267s user 0.008s sys 0.027s
W20250114 08:59:08.606808 21596 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.020s sys 0.025s
W20250114 08:59:09.046644 21668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.222s user 0.010s sys 0.066s
W20250114 08:59:09.053192 21553 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.336s user 0.015s sys 0.044s
W20250114 08:59:09.043787 23487 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.175s user 0.023s sys 0.017s
W20250114 08:59:09.038112 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 563537us
W20250114 08:59:09.087477 21596 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.361s user 0.023s sys 0.032s
W20250114 08:59:09.087179 21668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.262s user 0.014s sys 0.066s
W20250114 08:59:09.089540 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 1.36283e+06us
W20250114 08:59:09.089668 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 1.28313e+06us
W20250114 08:59:08.173202 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 475331us
W20250114 08:59:09.029629 21702 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.999s user 0.007s sys 0.006s
W20250114 08:59:08.608203 22925 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.049s sys 0.098s
W20250114 08:59:09.090989 21702 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.060s user 0.007s sys 0.007s
W20250114 08:59:09.087150 21550 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.325s user 0.008s sys 0.030s
W20250114 08:59:08.836339 22843 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.005s sys 0.024s
W20250114 08:59:08.426322 23257 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.018s sys 0.033s
W20250114 08:59:08.754109 23479 consensus_meta.cc:287] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.742s user 0.006s sys 0.005s
W20250114 08:59:08.608904 23491 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.001s sys 0.001s
W20250114 08:59:08.555830 23417 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.016s sys 0.021s
W20250114 08:59:08.899263 23500 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.000s sys 0.012s
W20250114 08:59:09.119935 23500 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.123s user 0.000s sys 0.016s
I20250114 08:59:09.116454 23430 consensus_queue.cc:1046] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20250114 08:59:08.836581 23444 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.006s sys 0.007s
W20250114 08:59:09.121131 23444 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.202s user 0.012s sys 0.007s
W20250114 08:59:08.980201 23483 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.792s user 0.009s sys 0.026s
W20250114 08:59:09.087150 21594 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.354s user 0.018s sys 0.021s
W20250114 08:59:08.808625 22270 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.006s user 0.045s sys 0.114s
W20250114 08:59:08.886844 22266 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.946s user 0.000s sys 0.011s
I20250114 08:59:09.107218 21596 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:08.936923 23482 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.002s sys 0.010s
I20250114 08:59:08.923309 23537 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.107687 23494 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:09.113992 23491 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.095s user 0.008s sys 0.006s
W20250114 08:59:09.089164 23487 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.220s user 0.024s sys 0.018s
I20250114 08:59:09.030534 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48992 (request call id 201) took 1189 ms. Trace:
I20250114 08:59:08.424624 23051 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.117774 21520 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:09.105357 22925 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.355s user 0.086s sys 0.131s
I20250114 08:59:09.118705 21697 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:09.108227 23298 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.068397 23490 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:09.090544 23309 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:09.113197 23442 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.130s user 0.002s sys 0.016s
W20250114 08:59:09.125396 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 1.39768e+06us
I20250114 08:59:09.221781 21625 raft_consensus.cc:1270] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:09.223233 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:09.224298 21623 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:09.129994 23500 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:08.534406 23449 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:09.117893 23417 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.413s user 0.022s sys 0.026s
W20250114 08:59:09.089051 23534 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.315s user 0.000s sys 0.001s
W20250114 08:59:09.124188 23483 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.945s user 0.010s sys 0.028s
W20250114 08:59:09.140225 21703 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.018s sys 0.000s
W20250114 08:59:09.110942 23257 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.363s user 0.030s sys 0.035s
I20250114 08:59:08.184531 23329 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.935s
W20250114 08:59:09.301647 21703 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.287s user 0.018s sys 0.000s
I20250114 08:59:09.166977 21622 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20250114 08:59:09.126452 22266 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.185s user 0.003s sys 0.011s
W20250114 08:59:09.126801 22270 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.324s user 0.064s sys 0.128s
I20250114 08:59:09.122182 23412 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:09.126804 23482 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.101s user 0.002s sys 0.015s
W20250114 08:59:09.146725 23513 consensus_meta.cc:287] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.114s user 0.001s sys 0.011s
W20250114 08:59:09.227705 21698 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.648s user 0.032s sys 0.053s
W20250114 08:59:09.326539 21698 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.747s user 0.032s sys 0.053s
W20250114 08:59:09.140048 23349 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.134s user 0.014s sys 0.005s
I20250114 08:59:09.230770 21701 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
W20250114 08:59:09.105427 22843 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.152s user 0.006s sys 0.025s
I20250114 08:59:09.134351 23444 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:09.296049 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 178385us
I20250114 08:59:09.226438 23487 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:09.122156 23479 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.207943 23430 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:09.221254 23442 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.238s user 0.002s sys 0.017s
W20250114 08:59:09.206502 22241 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.010s sys 0.013s
W20250114 08:59:09.248309 23421 consensus_meta.cc:287] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.194s user 0.005s sys 0.006s
I20250114 08:59:09.360644 23430 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:09.361097 23421 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:09.361327 23430 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.361702 23421 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.218003 23298 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.363765 23421 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.364085 23441 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:09.332682 23513 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.364663 23441 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:09.365460 21627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1307) took 3176 ms (client timeout 1803 ms). Trace:
W20250114 08:59:09.365859 21627 rpcz_store.cc:269] 0114 08:59:06.188993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.307704 (+118711us) service_pool.cc:225] Handling call
0114 08:59:09.365215 (+3057511us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54003b97d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:09.365367 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1391656320}
W20250114 08:59:09.237875 23431 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.351s user 0.029s sys 0.056s
I20250114 08:59:09.366734 23441 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.228200 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:09.305241 23232 raft_consensus.cc:2758] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250114 08:59:09.301110 22925 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:09.133674 23537 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:09.328794 23349 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.323s user 0.014s sys 0.007s
I20250114 08:59:09.332486 23449 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.366875 23513 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:09.214147 23051 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.367693 21621 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:09.331082 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:09.367825 23429 raft_consensus.cc:3049] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
W20250114 08:59:09.366921 23431 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.485s user 0.031s sys 0.059s
W20250114 08:59:09.368482 22241 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.852s user 0.011s sys 0.014s
I20250114 08:59:09.135591 22267 rpcz_store.cc:276] 0114 08:59:07.841199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.853457 (+ 12258us) service_pool.cc:225] Handling call
0114 08:59:09.029446 (+1175989us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:09.029762 (+ 316us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:09.030031 (+ 269us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:09.030075 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.030137 (+ 62us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:09.030185 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.030270 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.030454 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24098176}
W20250114 08:59:09.370785 21547 consensus_meta.cc:287] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.220s user 0.010s sys 0.002s
I20250114 08:59:09.364292 23421 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:09.386408 23441 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:09.330049 23433 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.327597 23490 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.372247 23430 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:09.227509 23309 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.383785 23183 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.355098 23345 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:09.296828 23500 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.386785 23378 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250114 08:59:09.416637 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 140) took 1380 ms. Trace:
I20250114 08:59:09.448751 21629 rpcz_store.cc:276] 0114 08:59:08.036258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:08.036484 (+ 226us) service_pool.cc:225] Handling call
0114 08:59:08.039026 (+ 2542us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:08.126955 (+ 87929us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.406065 (+1279110us) spinlock_profiling.cc:243] Waited 54.7 ms on lock 0x7b540022db54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:09.416563 (+ 10498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125835008}
W20250114 08:59:09.441792 21551 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.525s user 0.010s sys 0.040s
W20250114 08:59:09.417248 21519 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.015s sys 0.051s
I20250114 08:59:09.329448 23412 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.371508 23350 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:09.435698 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 104677us
I20250114 08:59:09.452513 21520 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:09.454576 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:09.455643 22239 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.987s user 0.028s sys 0.044s
W20250114 08:59:09.455900 22239 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.028s sys 0.044s
I20250114 08:59:09.338614 23444 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.424582 23429 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:09.428658 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 1.17078e+06us
I20250114 08:59:09.379084 23487 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.428750 23225 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:09.450466 21519 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.015s sys 0.051s
I20250114 08:59:09.375401 23479 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.436535 23435 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:09.459357 21520 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:09.450098 21551 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.534s user 0.011s sys 0.040s
W20250114 08:59:09.524890 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 68434.1us
I20250114 08:59:09.443084 23513 raft_consensus.cc:3049] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:09.517012 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1321) took 2026 ms. Trace:
I20250114 08:59:09.532691 21554 rpcz_store.cc:276] 0114 08:59:07.490279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.868767 (+378488us) service_pool.cc:225] Handling call
0114 08:59:07.869144 (+ 377us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:07.891395 (+ 22251us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:07.891424 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:07.907260 (+ 15836us) log.cc:844] Serialized 52 byte log entry
0114 08:59:09.052246 (+1144986us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:09.052376 (+ 130us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.052507 (+ 131us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:09.516709 (+464202us) raft_consensus.cc:1680] finished
0114 08:59:09.516737 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.516937 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10349,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":26740,"thread_start_us":1119542,"threads_started":1,"wal-append.queue_time_us":1533269}
W20250114 08:59:09.534633 21552 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.588s user 0.024s sys 0.062s
I20250114 08:59:09.436314 21547 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
W20250114 08:59:09.517469 23518 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.355s user 0.007s sys 0.007s
I20250114 08:59:09.446471 21695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1190) took 3400 ms. Trace:
W20250114 08:59:09.535105 21552 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.589s user 0.024s sys 0.062s
I20250114 08:59:09.461478 22925 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:09.457172 23537 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:09.514412 23549 log.cc:926] Time spent T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250114 08:59:09.440752 23421 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.515633 23449 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:09.518540 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:09.521347 23487 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:09.558369 21699 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.513s user 0.028s sys 0.069s
W20250114 08:59:09.615008 21699 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.570s user 0.029s sys 0.070s
I20250114 08:59:09.462110 23535 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:09.446177 23183 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.549116 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:09.628276 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 119) took 1572 ms. Trace:
I20250114 08:59:09.628795 21554 rpcz_store.cc:276] 0114 08:59:08.055594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.548735 (+1493141us) service_pool.cc:225] Handling call
0114 08:59:09.628192 (+ 79457us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:09.559389 21547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 118) took 1580 ms (client timeout 2094 ms). Trace:
I20250114 08:59:09.532585 23513 raft_consensus.cc:738] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:09.440675 23433 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.446499 23441 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:09.625453 21669 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.002s sys 0.033s
I20250114 08:59:09.440834 23490 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.522742 23350 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:09.636920 21669 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.593s user 0.002s sys 0.034s
I20250114 08:59:09.649704 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:09.651878 21554 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:09.615064 23449 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:09.615682 23309 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:09.657483 21548 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.024s sys 0.041s
I20250114 08:59:09.657706 21626 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 08:59:09.657871 21548 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.024s sys 0.041s
I20250114 08:59:09.684762 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1320) took 2203 ms. Trace:
I20250114 08:59:09.519243 23429 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:09.724984 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1294) took 3540 ms. Trace:
W20250114 08:59:09.723902 21555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1237) took 2891 ms (client timeout 1905 ms). Trace:
W20250114 08:59:09.726053 21555 rpcz_store.cc:269] 0114 08:59:06.813956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.159049 (+345093us) service_pool.cc:225] Handling call
0114 08:59:09.705645 (+2546596us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x7b54003a16d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:09.705813 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":170532608}
I20250114 08:59:09.727417 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:09.721678 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1319) took 2445 ms. Trace:
I20250114 08:59:09.725863 21624 rpcz_store.cc:276] 0114 08:59:06.184845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.766547 (+581702us) service_pool.cc:225] Handling call
0114 08:59:06.959939 (+193392us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:06.963472 (+ 3533us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.724783 (+2761311us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54003bb0d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:09.724929 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443947904}
W20250114 08:59:09.731248 21553 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.038s sys 0.068s
W20250114 08:59:09.731657 21553 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.038s sys 0.068s
I20250114 08:59:09.732759 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1315) took 2668 ms. Trace:
I20250114 08:59:09.728371 21550 rpcz_store.cc:276] 0114 08:59:07.276662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.754041 (+477379us) service_pool.cc:225] Handling call
0114 08:59:07.761344 (+ 7303us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:07.761714 (+ 370us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:07.761751 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:07.762211 (+ 460us) log.cc:844] Serialized 26 byte log entry
0114 08:59:09.096206 (+1333995us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:09.096249 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.096319 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:09.721372 (+625053us) raft_consensus.cc:1680] finished
0114 08:59:09.721394 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.721601 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4426,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":32801,"thread_start_us":1333581,"threads_started":1,"wal-append.queue_time_us":1617176}
I20250114 08:59:09.733251 21553 rpcz_store.cc:276] 0114 08:59:07.064033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.704097 (+640064us) service_pool.cc:225] Handling call
0114 08:59:07.706015 (+ 1918us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:07.706206 (+ 191us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 08:59:09.087383 (+1381177us) log.cc:844] Serialized 14 byte log entry
0114 08:59:09.732289 (+644906us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.732365 (+ 76us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:09.732426 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.732534 (+ 108us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.732691 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1686693,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":4547,"thread_start_us":2014897,"threads_started":2}
I20250114 08:59:09.559628 21695 rpcz_store.cc:276] 0114 08:59:06.036077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.122442 (+ 86365us) service_pool.cc:225] Handling call
0114 08:59:06.188698 (+ 66256us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:06.960412 (+771714us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.404956 (+2444544us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b5400346a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:09.435906 (+ 30950us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989715712}
W20250114 08:59:09.634084 21547 rpcz_store.cc:269] 0114 08:59:07.978544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:08.018553 (+ 40009us) service_pool.cc:225] Handling call
0114 08:59:09.551751 (+1533198us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:09.686914 23565 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.112s user 0.001s sys 0.001s
I20250114 08:59:09.657824 23051 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:09.740739 21702 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.008s sys 0.014s
W20250114 08:59:09.741241 21702 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.008s sys 0.014s
I20250114 08:59:09.741781 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1501) took 1721 ms. Trace:
W20250114 08:59:09.725225 23564 log.cc:926] Time spent T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.347s user 0.000s sys 0.001s
W20250114 08:59:09.742388 21549 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.010s sys 0.005s
W20250114 08:59:09.743489 21668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.008s sys 0.016s
W20250114 08:59:09.743431 21593 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.010s sys 0.014s
W20250114 08:59:09.745694 21668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.009s sys 0.016s
W20250114 08:59:09.744103 22240 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.713s user 0.000s sys 0.014s
I20250114 08:59:09.742142 21702 rpcz_store.cc:276] 0114 08:59:08.019973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:08.020112 (+ 139us) service_pool.cc:225] Handling call
0114 08:59:08.022347 (+ 2235us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:08.022539 (+ 192us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:08.030712 (+ 8173us) log.cc:844] Serialized 14 byte log entry
0114 08:59:09.741504 (+1710792us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.741550 (+ 46us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:09.741593 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.741653 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.741744 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5577,"prepare.queue_time_us":8018,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":281607,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":4964,"thread_start_us":1710355,"threads_started":2,"wal-append.queue_time_us":1277908}
W20250114 08:59:09.746483 22240 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.000s sys 0.014s
W20250114 08:59:09.746345 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 634944us
I20250114 08:59:09.747093 21553 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:09.750025 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:09.750412 21553 raft_consensus.cc:604] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:09.750797 21553 raft_consensus.cc:612] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250114 08:59:09.746168 21593 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.010s sys 0.014s
I20250114 08:59:09.752074 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:09.745730 21549 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.010s sys 0.005s
I20250114 08:59:09.753564 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 115) took 2384 ms. Trace:
I20250114 08:59:09.724434 23519 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.755852 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:09.753969 21549 rpcz_store.cc:276] 0114 08:59:07.368890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.767297 (+398407us) service_pool.cc:225] Handling call
0114 08:59:07.803318 (+ 36021us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:07.803499 (+ 181us) raft_consensus.cc:1529] Early marking committed up to index 5
0114 08:59:09.052825 (+1249326us) log.cc:844] Serialized 14 byte log entry
0114 08:59:09.753256 (+700431us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.753307 (+ 51us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:09.753347 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.753406 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.753510 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1618554,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":79,"thread_start_us":1928264,"threads_started":2}
W20250114 08:59:09.805222 21594 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.008s sys 0.014s
W20250114 08:59:09.809767 21594 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.008s sys 0.014s
I20250114 08:59:09.752748 21550 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:09.807515 21700 raft_consensus.cc:2388] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:09.737983 23518 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:09.808249 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 116) took 2171 ms. Trace:
I20250114 08:59:09.811451 21555 rpcz_store.cc:276] 0114 08:59:07.637007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.745449 (+2108442us) service_pool.cc:225] Handling call
0114 08:59:09.807729 (+ 62280us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:09.807936 (+ 207us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:09.807971 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.808010 (+ 39us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:09.808036 (+ 26us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.808085 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.808191 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:09.808624 21554 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
W20250114 08:59:09.751116 23560 log.cc:926] Time spent T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.334s user 0.000s sys 0.001s
W20250114 08:59:09.755456 23562 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.325s user 0.001s sys 0.001s
I20250114 08:59:09.752559 23429 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:59:09.817636 21670 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.004s sys 0.015s
W20250114 08:59:09.829058 21670 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.798s user 0.004s sys 0.015s
I20250114 08:59:09.832741 21555 raft_consensus.cc:1270] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:09.822392 22885 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.835671 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:09.835922 22885 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.836432 22271 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:09.756685 23449 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:09.837568 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 378276us
I20250114 08:59:09.826411 23385 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.838567 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:09.840528 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:09.810438 21594 connection.cc:232] RPC call timeout handler was delayed by -1.0473688770000535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
I20250114 08:59:09.843878 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:09.847508 21518 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:09.849459 23547 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.850132 23547 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.852491 23569 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:09.813066 23568 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:09.854071 23541 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
I20250114 08:59:09.853984 23568 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.855428 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1491) took 2435 ms. Trace:
I20250114 08:59:09.810550 23051 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.864105 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:09.875327 21700 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 5
I20250114 08:59:09.857586 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:09.876549 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:09.815591 23535 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:09.823496 23479 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:09.896435 21668 connection.cc:232] RPC call timeout handler was delayed by -3.5010644309999179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20250114 08:59:09.855798 21698 rpcz_store.cc:276] 0114 08:59:07.420317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.420905 (+ 588us) service_pool.cc:225] Handling call
0114 08:59:07.421225 (+ 320us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:07.486445 (+ 65220us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:07.486503 (+ 58us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:07.495003 (+ 8500us) log.cc:844] Serialized 26 byte log entry
0114 08:59:09.326969 (+1831966us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:09.327014 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.327113 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:09.855216 (+528103us) raft_consensus.cc:1680] finished
0114 08:59:09.855235 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.855366 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114698,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":89,"thread_start_us":1786729,"threads_started":2,"wal-append.queue_time_us":2060610}
I20250114 08:59:09.926177 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:09.936517 21668 connection.cc:232] RPC call timeout handler was delayed by -3.4750598929999796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
I20250114 08:59:09.823071 23575 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:09.816411 23487 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:09.928218 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 113308us
I20250114 08:59:09.932433 21623 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250114 08:59:09.932943 23542 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.360s user 0.000s sys 0.001s
W20250114 08:59:09.933398 23554 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20250114 08:59:09.741676 23513 consensus_queue.cc:260] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.934286 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:09.894963 23444 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.939720 21630 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:09.841693 23385 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:09.946586 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:09.837795 23430 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:09.857131 23309 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:09.837985 23225 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:09.852339 23547 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:09.858112 23510 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.956046 23516 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.880571 23476 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.858557 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:09.936990 21668 connection.cc:232] RPC call timeout handler was delayed by -3.4549751979999428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20250114 08:59:09.824386 23518 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.928972 23500 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.880951 23412 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:09.856380 23568 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.815567 23519 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.974388 21623 raft_consensus.cc:1270] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:59:09.974653 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:09.998477 21700 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:09.958011 23483 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:09.946532 22885 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.955552 23183 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.938266 23051 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.858268 23449 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:09.857007 23580 consensus_queue.cc:1046] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20250114 08:59:09.830796 23570 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.134s
I20250114 08:59:09.945022 23575 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.946153 23535 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:09.962878 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 117) took 2250 ms. Trace:
I20250114 08:59:10.086499 22275 raft_consensus.cc:1235] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:10.061524 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.026636 23441 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:09.959827 23392 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:10.313778 23536 log.cc:926] Time spent T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.504s user 0.000s sys 0.000s
I20250114 08:59:09.961791 23300 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:10.077350 23559 log.cc:926] Time spent T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.459s user 0.000s sys 0.001s
I20250114 08:59:10.314109 23417 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.318359 23350 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.314831 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:09.973651 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.316092 23518 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.316681 21629 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:10.026541 23225 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.429018 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:09.973295 23309 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:10.430188 23309 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:10.004709 23433 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:10.310153 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 471699us
I20250114 08:59:10.431535 23518 consensus_queue.cc:1046] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
W20250114 08:59:10.311981 21668 leader_election.cc:336] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.669s (SENT)
I20250114 08:59:10.097108 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:10.311720 21550 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:59:10.087343 21554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:10.074586 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:10.272022 23557 log.cc:926] Time spent T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.518s user 0.000s sys 0.001s
W20250114 08:59:10.300706 23553 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.344s user 0.000s sys 0.001s
W20250114 08:59:10.246572 23552 log.cc:926] Time spent T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.507s user 0.000s sys 0.001s
W20250114 08:59:10.293013 23558 log.cc:926] Time spent T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.457s user 0.000s sys 0.000s
W20250114 08:59:10.086701 23566 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20250114 08:59:10.335927 23051 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.336040 23183 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:10.107044 23562 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250114 08:59:10.336689 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.336534 23516 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.004407 23430 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.361560 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1212) took 4011 ms. Trace:
I20250114 08:59:10.361552 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:09.960732 23487 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.308211 23440 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:09.966099 23421 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.278510 23576 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:10.260756 22270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 153) took 2447 ms. Trace:
I20250114 08:59:10.232905 22266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 150) took 4205 ms. Trace:
W20250114 08:59:10.422361 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 52098.5us
I20250114 08:59:10.272246 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.433830 21554 raft_consensus.cc:604] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:10.432941 21704 raft_consensus.cc:2371] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5. Re-sending same reply.
I20250114 08:59:09.997920 23431 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.372s
I20250114 08:59:10.278689 23571 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:10.336045 23574 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.208715 23592 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.441845 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:10.510028 21594 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.019s sys 0.109s
W20250114 08:59:10.510524 21594 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.019s sys 0.109s
I20250114 08:59:10.329080 21695 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
W20250114 08:59:10.441838 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 129892us
W20250114 08:59:10.510774 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 565821us
W20250114 08:59:10.511162 21668 leader_election.cc:336] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.686s (SENT)
I20250114 08:59:10.511605 21668 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:10.360749 23500 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.360697 23493 raft_consensus.cc:3049] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:10.420754 23483 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.479529 21621 rpcz_store.cc:276] 0114 08:59:06.349713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:06.804282 (+454569us) service_pool.cc:225] Handling call
0114 08:59:09.367495 (+2563213us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:09.423955 (+ 56460us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:10.361324 (+937369us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54003b97d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:10.361496 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5882205952}
I20250114 08:59:10.361274 23568 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:10.524500 21668 leader_election.cc:336] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.720s (SENT)
I20250114 08:59:10.322798 23476 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.491562 23183 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:10.499784 23583 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
I20250114 08:59:10.457489 23546 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.500424 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1490) took 3080 ms. Trace:
I20250114 08:59:10.536834 21701 rpcz_store.cc:276] 0114 08:59:07.419954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.420452 (+ 498us) service_pool.cc:225] Handling call
0114 08:59:07.420733 (+ 281us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:09.332190 (+1911457us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:10.500177 (+1167987us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54001653d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:10.500368 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4162109184}
I20250114 08:59:10.536823 23546 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.538297 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:10.501607 21624 raft_consensus.cc:2405] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:10.362394 23392 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.456697 23441 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:10.375365 23300 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:10.514540 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 72836.7us
I20250114 08:59:10.514678 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:10.515619 21521 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.050s sys 0.095s
I20250114 08:59:10.515990 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:10.541699 21521 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.050s sys 0.095s
I20250114 08:59:10.428637 23417 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:10.506436 23585 log.cc:926] Time spent T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.291s user 0.000s sys 0.001s
I20250114 08:59:10.500988 23433 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.546022 21704 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 6
W20250114 08:59:10.546159 21627 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.011s sys 0.016s
W20250114 08:59:10.546663 21627 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.689s user 0.011s sys 0.016s
I20250114 08:59:10.461112 23350 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250114 08:59:10.424165 23471 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:10.326828 23510 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:10.532407 21668 leader_election.cc:336] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.846s (SENT)
W20250114 08:59:10.533284 21696 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.006s sys 0.010s
I20250114 08:59:10.516670 23232 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:10.534878 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1468) took 4551 ms. Trace:
I20250114 08:59:10.535598 23183 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.542781 22239 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:10.319749 21547 rpcz_store.cc:276] 0114 08:59:07.712862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.761451 (+2048589us) service_pool.cc:225] Handling call
0114 08:59:09.837696 (+ 76245us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:09.880319 (+ 42623us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:09.893481 (+ 13162us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.935082 (+ 41601us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:09.939067 (+ 3985us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.949689 (+ 10622us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.960644 (+ 10955us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:10.520598 23576 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:10.554386 21668 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:10.554908 21696 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.007s sys 0.010s
I20250114 08:59:10.555112 21703 rpcz_store.cc:276] 0114 08:59:05.983335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:05.983452 (+ 117us) service_pool.cc:225] Handling call
0114 08:59:06.004235 (+ 20783us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:06.015038 (+ 10803us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:06.015072 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:06.049196 (+ 34124us) log.cc:844] Serialized 26 byte log entry
0114 08:59:08.014638 (+1965442us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b18002e9120. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:59:09.302230 (+1287592us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:09.302289 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.302405 (+ 116us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:10.534560 (+1232155us) raft_consensus.cc:1680] finished
0114 08:59:10.534579 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:10.534796 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16559,"raft.run_cpu_time_us":640,"raft.run_wall_time_us":38958,"spinlock_wait_cycles":54063360,"thread_start_us":1287373,"threads_started":1,"wal-append.queue_time_us":3399451}
I20250114 08:59:10.511792 23440 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.526587 23493 raft_consensus.cc:738] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:10.564877 23421 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:10.565374 23309 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.566471 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:10.567344 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1502) took 2524 ms. Trace:
W20250114 08:59:10.573995 21668 leader_election.cc:336] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.886s (SENT)
I20250114 08:59:10.520762 23500 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.545719 23444 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:10.577960 21668 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:10.582357 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:10.577766 21699 rpcz_store.cc:276] 0114 08:59:08.043424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:08.043609 (+ 185us) service_pool.cc:225] Handling call
0114 08:59:08.044040 (+ 431us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:08.044651 (+ 611us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:08.044686 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:08.045200 (+ 514us) log.cc:844] Serialized 26 byte log entry
0114 08:59:09.615459 (+1570259us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:09.615501 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.615571 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:10.567076 (+951505us) raft_consensus.cc:1680] finished
0114 08:59:10.567098 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:10.567269 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151,"raft.queue_time_us":7349,"spinlock_wait_cycles":2527488,"thread_start_us":1569938,"threads_started":1,"wal-append.queue_time_us":1786692}
I20250114 08:59:10.586196 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:10.581714 23232 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:10.592339 21703 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:10.602185 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:10.572677 23350 consensus_queue.cc:1225] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:10.603253 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.635053 21520 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:10.575220 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:10.547883 23441 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:10.636142 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 62189.3us
W20250114 08:59:10.651912 21668 leader_election.cc:336] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.936s (SENT)
I20250114 08:59:10.652410 21668 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:10.596419 21704 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:10.642474 23599 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:10.643471 23580 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:10.551154 23433 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:10.646214 21670 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.044s sys 0.064s
W20250114 08:59:10.705859 21670 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.044s sys 0.065s
I20250114 08:59:10.517616 23592 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.575440 23597 consensus_queue.cc:1035] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.157s
W20250114 08:59:10.715659 23595 log.cc:926] Time spent T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20250114 08:59:10.515705 23571 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.523111 23574 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.605937 23444 raft_consensus.cc:695] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:10.557199 23471 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.584969 23422 consensus_queue.cc:1035] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20250114 08:59:10.692332 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:10.729498 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 94533.3us
I20250114 08:59:10.726423 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1308) took 3981 ms. Trace:
I20250114 08:59:10.731114 21551 rpcz_store.cc:276] 0114 08:59:06.738513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.639119 (+900606us) service_pool.cc:225] Handling call
0114 08:59:07.659116 (+ 19997us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:07.870676 (+211560us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:07.870706 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:07.880947 (+ 10241us) log.cc:844] Serialized 52 byte log entry
0114 08:59:09.529135 (+1648188us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:09.529192 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:10.705910 (+1176718us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400323054. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:10.705971 (+ 61us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:10.706019 (+ 48us) raft_consensus.cc:1680] finished
0114 08:59:10.706032 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:10.719970 (+ 13938us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6766,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4510,"spinlock_wait_cycles":485648256,"thread_start_us":1612797,"threads_started":1,"wal-append.queue_time_us":1929657}
I20250114 08:59:10.650574 23535 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.693646 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:10.693681 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 129184us
I20250114 08:59:10.696002 23449 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.743844 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.744659 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.745879 21701 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:10.716307 23593 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:10.546341 23568 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:10.760263 21697 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1499) took 2812 ms (client timeout 2149 ms). Trace:
W20250114 08:59:10.760818 21697 rpcz_store.cc:269] 0114 08:59:07.948061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.948308 (+ 247us) service_pool.cc:225] Handling call
0114 08:59:10.759925 (+2811617us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400094554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:10.760133 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2687933312}
I20250114 08:59:10.762317 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.763149 22271 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:59:10.763449 23594 log.cc:926] Time spent T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
I20250114 08:59:10.718961 23570 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:10.631440 23575 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:10.688347 22241 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.024s sys 0.044s
I20250114 08:59:10.656957 23232 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:10.782696 23573 log.cc:926] Time spent T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.261s user 0.000s sys 0.001s
I20250114 08:59:10.782817 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.654443 23412 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:59:10.726351 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 182559us
I20250114 08:59:10.655444 23350 consensus_queue.cc:1035] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20250114 08:59:10.738720 21704 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:10.715570 23580 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.750685 23577 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250114 08:59:10.752116 23592 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:10.491847 22270 rpcz_store.cc:276] 0114 08:59:07.800290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.800892 (+ 602us) service_pool.cc:225] Handling call
0114 08:59:07.801328 (+ 436us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:07.801728 (+ 400us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:07.801765 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:07.802236 (+ 471us) log.cc:844] Serialized 26 byte log entry
0114 08:59:09.329472 (+1527236us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:09.335937 (+ 6465us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.355511 (+ 19574us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:10.163312 (+807801us) raft_consensus.cc:1680] finished
0114 08:59:10.183921 (+ 20609us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:10.241900 (+ 57979us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1822,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":14917,"thread_start_us":1505639,"threads_started":1,"wal-append.queue_time_us":1626917}
I20250114 08:59:10.752413 23430 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.757303 21699 raft_consensus.cc:2405] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:10.714612 23587 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 08:59:10.769318 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 225202us
I20250114 08:59:10.754348 23441 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.495091 22266 rpcz_store.cc:276] 0114 08:59:05.981014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:05.993337 (+ 12323us) service_pool.cc:225] Handling call
0114 08:59:07.923852 (+1930515us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:07.940505 (+ 16653us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:07.940783 (+ 278us) raft_consensus.cc:1529] Early marking committed up to index 7
0114 08:59:09.326559 (+1385776us) log.cc:844] Serialized 14 byte log entry
0114 08:59:09.959996 (+633437us) log.cc:844] Serialized 14 byte log entry
0114 08:59:10.004866 (+ 44870us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:10.026854 (+ 21988us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:10.075430 (+ 48576us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:10.089345 (+ 13915us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:10.164224 (+ 74879us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70768,"prepare.queue_time_us":1684803,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":304749,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":77112,"spinlock_wait_cycles":39168000,"thread_start_us":1888938,"threads_started":3,"wal-append.queue_time_us":1174484}
I20250114 08:59:10.771451 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:10.807979 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.713625 21703 raft_consensus.cc:2405] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250114 08:59:10.776257 23600 log.cc:926] Time spent T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20250114 08:59:10.777673 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.809546 21697 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:10.810680 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.706166 23493 consensus_queue.cc:260] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:10.782526 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:10.690073 23494 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.518s user 0.022s sys 0.004s
I20250114 08:59:10.785238 21594 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:10.798108 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 157068us
I20250114 08:59:10.800525 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:10.800659 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 129347us
W20250114 08:59:10.779765 22241 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.673s user 0.025s sys 0.044s
I20250114 08:59:10.752915 23516 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:10.808918 21704 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:10.817189 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:10.784940 23483 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:10.779014 23593 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:10.753643 23597 consensus_queue.cc:1046] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250114 08:59:10.781350 23568 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.705531 23588 consensus_queue.cc:1035] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.228s
I20250114 08:59:10.964143 23471 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.892447 23500 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:10.928225 23385 consensus_meta.cc:287] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.929s user 0.009s sys 0.004s
W20250114 08:59:10.962666 23550 log.cc:926] Time spent T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 1.533s user 0.001s sys 0.001s
I20250114 08:59:10.883293 23487 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.896735 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.896365 23574 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:10.908159 23298 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:10.867552 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 44700us
W20250114 08:59:11.028542 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 160374us
I20250114 08:59:11.060052 21521 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:09.685293 21548 rpcz_store.cc:276] 0114 08:59:07.481537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.863825 (+382288us) service_pool.cc:225] Handling call
0114 08:59:07.864142 (+ 317us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:07.864328 (+ 186us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 08:59:09.057236 (+1192908us) log.cc:844] Serialized 14 byte log entry
0114 08:59:09.684284 (+627048us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:09.684359 (+ 75us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:09.684418 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:09.684535 (+ 117us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:09.684688 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25600,"prepare.queue_time_us":1576923,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":3010,"raft.queue_time_us":25846,"thread_start_us":1759290,"threads_started":2}
I20250114 08:59:11.061026 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:10.821977 23599 raft_consensus.cc:695] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:11.061641 23385 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.062330 23599 consensus_queue.cc:237] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.063174 23358 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:11.063701 23358 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.064265 23358 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:11.063788 23385 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.871107 21701 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:10.902624 23610 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:11.067569 23610 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.893723 23392 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.936579 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1227) took 1090 ms. Trace:
I20250114 08:59:11.067569 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:11.068282 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:10.933588 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.068974 22270 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:10.902065 23309 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:11.070096 23309 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:59:10.910948 21626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1326) took 1743 ms (client timeout 1999 ms). Trace:
W20250114 08:59:11.070737 21626 rpcz_store.cc:269] 0114 08:59:09.167884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.168033 (+ 149us) service_pool.cc:225] Handling call
0114 08:59:10.910674 (+1742641us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54002f9754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:10.910844 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":652590848}
I20250114 08:59:10.892179 23571 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:10.884824 23510 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.932694 23580 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.896648 23444 consensus_queue.cc:237] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.075090 23482 raft_consensus.cc:3049] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:11.075592 23482 raft_consensus.cc:738] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:11.068604 21622 rpcz_store.cc:276] 0114 08:59:09.845832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.845977 (+ 145us) service_pool.cc:225] Handling call
0114 08:59:09.846394 (+ 417us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:09.930025 (+ 83631us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:09.930069 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:09.930752 (+ 683us) log.cc:844] Serialized 52 byte log entry
0114 08:59:10.214512 (+283760us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:10.214570 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:10.214678 (+ 108us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:10.936318 (+721640us) raft_consensus.cc:1680] finished
0114 08:59:10.936339 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:10.936514 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":854,"raft.queue_time_us":50744,"raft.run_cpu_time_us":426,"raft.run_wall_time_us":28661,"thread_start_us":246278,"threads_started":1,"wal-append.queue_time_us":631992}
W20250114 08:59:11.041543 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 1.09591e+06us
W20250114 08:59:11.079620 22241 connection.cc:232] RPC call timeout handler was delayed by -1.1889447390000214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
I20250114 08:59:11.040933 22925 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.080997 22925 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:11.080400 22241 connection.cc:232] RPC call timeout handler was delayed by -1.1691447389999894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20250114 08:59:11.048494 23593 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:11.083391 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:11.089170 23225 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:11.089673 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.091830 23580 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:11.092262 23580 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.092761 23580 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:10.905850 23433 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.785293 23570 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:10.826798 23490 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:10.866236 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.101789 21554 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:10.814324 23300 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:10.806123 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:11.032922 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 302030us
I20250114 08:59:11.104295 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:11.034701 21596 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:11.035172 21668 connection.cc:232] RPC call timeout handler was delayed by -3.3173993839999412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
I20250114 08:59:10.978312 23440 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:11.035670 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 250515us
I20250114 08:59:11.120311 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:11.120729 21696 raft_consensus.cc:2405] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250114 08:59:10.978802 23429 consensus_meta.cc:287] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.145s user 0.006s sys 0.008s
I20250114 08:59:11.037607 21697 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:11.125500 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.126327 21697 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:11.127451 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.038695 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.040607 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1324) took 3503 ms. Trace:
I20250114 08:59:10.978808 23474 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:11.052760 23597 raft_consensus.cc:3049] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:11.128693 21552 rpcz_store.cc:276] 0114 08:59:07.536892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.927660 (+390768us) service_pool.cc:225] Handling call
0114 08:59:07.946020 (+ 18360us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:07.946269 (+ 249us) raft_consensus.cc:1529] Early marking committed up to index 5
0114 08:59:10.914368 (+2968099us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b18002712a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63f25808c 00007fa63c676bfc 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c691305 00007fa642fa2107 00007fa63f5d78f7 00007fa63f5e700f
0114 08:59:10.914596 (+ 228us) log.cc:844] Serialized 14 byte log entry
0114 08:59:11.040205 (+125609us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:11.040258 (+ 53us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:11.040318 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.040382 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:11.040519 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1522003,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":86,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":5983,"spinlock_wait_cycles":29931392,"thread_start_us":1738475,"threads_started":2,"wal-append.queue_time_us":136150}
W20250114 08:59:10.934217 23538 log.cc:926] Time spent T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 1.499s user 0.005s sys 0.000s
I20250114 08:59:11.042047 23609 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.131855 23609 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:11.132155 23610 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.132288 23609 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:11.069038 22266 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:59:11.070253 21521 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:10.826299 23476 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:11.099023 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 143) took 1261 ms. Trace:
I20250114 08:59:10.818948 23417 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:11.119347 21668 connection.cc:232] RPC call timeout handler was delayed by -3.1667636759999596s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
W20250114 08:59:11.137806 21668 connection.cc:232] RPC call timeout handler was delayed by -3.1515898940000397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
W20250114 08:59:11.138114 21668 connection.cc:232] RPC call timeout handler was delayed by -3.0699910459999273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2565s for a response
W20250114 08:59:11.138293 21668 connection.cc:232] RPC call timeout handler was delayed by -3.0677299139999832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3103s for a response
W20250114 08:59:11.138621 21668 connection.cc:232] RPC call timeout handler was delayed by -2.9880910459999086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2656s for a response
W20250114 08:59:11.138816 21668 connection.cc:232] RPC call timeout handler was delayed by -2.9646910459998708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2682s for a response
W20250114 08:59:11.138968 21668 connection.cc:232] RPC call timeout handler was delayed by -2.8743741659999387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25910000000000005s for a response
I20250114 08:59:11.034471 21625 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
W20250114 08:59:11.121922 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 87312.1us
I20250114 08:59:11.128765 21704 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 1.
I20250114 08:59:11.053251 23611 consensus_queue.cc:1046] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:11.084326 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 306) took 1211 ms. Trace:
I20250114 08:59:11.128175 21697 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
W20250114 08:59:11.135618 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 65447.3us
I20250114 08:59:11.136265 21629 rpcz_store.cc:276] 0114 08:59:09.837521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.838227 (+ 706us) service_pool.cc:225] Handling call
0114 08:59:09.893623 (+ 55396us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:10.424291 (+530668us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.098765 (+674474us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b5400290ed4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:11.098961 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":862970112}
I20250114 08:59:11.142652 22240 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:11.129945 23225 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:11.159684 21521 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.124905 23433 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:11.161046 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.170210 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.161823 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.130079 23597 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:11.125038 21695 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
W20250114 08:59:11.146471 21668 connection.cc:232] RPC call timeout handler was delayed by -2.7889822070000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
I20250114 08:59:11.087257 23482 consensus_queue.cc:260] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.145421 23440 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:11.173121 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.163113 23588 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:11.177898 21668 connection.cc:232] RPC call timeout handler was delayed by -2.7790822069999876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20250114 08:59:11.181337 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.189328 21552 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:59:11.189950 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.144582 23392 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.184181 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:11.157774 23512 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.185597 21521 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.186830 21697 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
W20250114 08:59:11.170989 21520 connection.cc:232] RPC call timeout handler was delayed by -2.4881730649999554s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20250114 08:59:11.168704 23569 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.163533 23611 consensus_queue.cc:1225] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:11.160739 23309 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.201867 21555 raft_consensus.cc:1270] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250114 08:59:11.190996 21668 connection.cc:232] RPC call timeout handler was delayed by -2.7529822069999454s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1572s for a response
I20250114 08:59:11.168869 23225 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:11.193583 21704 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:11.144831 23614 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.196213 21521 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.197579 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.206056 21521 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:11.200206 21549 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:11.206310 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.207697 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.207592 23225 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.200739 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.201496 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:11.202450 21668 connection.cc:232] RPC call timeout handler was delayed by -2.6633952709998994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2514s for a response
I20250114 08:59:11.158493 22267 rpcz_store.cc:276] 0114 08:59:09.872831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.926972 (+ 54141us) service_pool.cc:225] Handling call
0114 08:59:11.083218 (+1156246us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:11.084236 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10658176}
I20250114 08:59:11.201890 23512 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.203681 23309 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.163228 23510 consensus_queue.cc:1035] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 08:59:11.186617 23440 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.215384 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.214932 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.231328 21704 raft_consensus.cc:1808] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
W20250114 08:59:11.231446 21594 connection.cc:232] RPC call timeout handler was delayed by -1.5588014809999322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20250114 08:59:11.166922 23617 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.188400 23400 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:11.217921 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.214931 23615 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:11.172394 23592 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20250114 08:59:11.219966 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.220573 21550 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:11.220963 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.192564 23588 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.198520 23569 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:11.203284 23437 raft_consensus.cc:2798] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 08:59:11.208292 23480 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:11.223119 21548 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:11.244549 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.245306 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.245409 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.224794 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:11.226027 23570 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:11.227212 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:11.188081 23613 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.231834 21704 raft_consensus.cc:2405] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
W20250114 08:59:11.211210 21668 connection.cc:232] RPC call timeout handler was delayed by -2.3591014070000256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2473s for a response
W20250114 08:59:11.249800 21668 connection.cc:232] RPC call timeout handler was delayed by -2.2001635919999671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
W20250114 08:59:11.250114 21668 connection.cc:232] RPC call timeout handler was delayed by -2.19948220699996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20250114 08:59:11.250288 21668 connection.cc:232] RPC call timeout handler was delayed by -2.0014822069999809s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24070000000000003s for a response
W20250114 08:59:11.250442 21668 connection.cc:232] RPC call timeout handler was delayed by -1.5894621709999228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.37650000000000006s for a response
W20250114 08:59:11.250592 21668 connection.cc:232] RPC call timeout handler was delayed by -1.3777822069999957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31000000000000005s for a response
W20250114 08:59:11.250746 21668 connection.cc:232] RPC call timeout handler was delayed by -1.3375594289999526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20250114 08:59:11.234246 23309 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.234728 23440 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:11.239645 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.254853 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.256832 21622 raft_consensus.cc:2388] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:11.240036 21555 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:11.257004 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.211611 21697 raft_consensus.cc:2405] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:11.221318 23599 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:11.197644 23392 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.225135 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.243506 23569 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:11.246896 21625 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:11.245076 21629 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:11.248294 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.242300 23615 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.237879 23400 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.236696 23617 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.258376 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.242018 23487 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.306449 23487 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.257751 21547 raft_consensus.cc:2388] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:11.253916 21668 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.532s (SENT) [suppressed 6 similar messages]
I20250114 08:59:11.308964 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.244328 23437 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.310884 21550 raft_consensus.cc:2388] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:11.309121 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 55223.9us
I20250114 08:59:11.216522 23614 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.307880 21551 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:11.272480 21695 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:11.259460 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.311661 21553 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:11.295025 21700 raft_consensus.cc:2405] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:11.254928 21521 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:11.267845 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.279929 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:11.316345 21668 leader_election.cc:336] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.572s (SENT)
W20250114 08:59:11.316694 23539 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 1.631s user 0.001s sys 0.002s
I20250114 08:59:11.320348 21668 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.320984 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:11.295490 21546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1224) took 4487 ms (client timeout 2222 ms). Trace:
I20250114 08:59:11.319732 21552 raft_consensus.cc:2405] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:11.322309 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.313886 23437 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:11.315744 23615 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:11.324905 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.327067 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.327967 21553 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20250114 08:59:11.323956 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 69141.7us
W20250114 08:59:11.325631 21668 leader_election.cc:336] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.857s (SENT)
W20250114 08:59:11.330014 21668 leader_election.cc:336] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.013s (SENT)
I20250114 08:59:11.318949 23479 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.313537 22270 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.331033 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.330402 21668 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.325315 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.339684 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.275642 23358 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.329061 23533 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:11.344906 23512 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:11.345512 23624 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20250114 08:59:11.326959 23309 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:11.347764 23519 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:11.352802 23617 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.337711 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.341686 21555 raft_consensus.cc:2388] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:11.258330 23613 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.358240 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.359465 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.360023 21553 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:11.313863 23400 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:11.342801 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:11.360332 21668 leader_election.cc:336] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.139s (SENT)
I20250114 08:59:11.361023 21668 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.355729 23588 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.360842 21552 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:11.363816 21704 raft_consensus.cc:2405] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:11.330960 23599 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.364915 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.366508 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:11.324203 21546 rpcz_store.cc:269] 0114 08:59:06.810581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:07.393459 (+582878us) service_pool.cc:225] Handling call
0114 08:59:11.295163 (+3901704us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b540031a954. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:11.295375 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4069508352}
W20250114 08:59:11.373669 21668 leader_election.cc:336] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.222s (SENT)
I20250114 08:59:11.382395 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.384192 21668 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.358474 23474 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:11.373374 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.389379 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.390697 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.391538 21704 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:11.392923 21521 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.393548 21555 raft_consensus.cc:1235] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 08:59:11.365860 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.340952 23479 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.375291 23588 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.374822 23472 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:11.401362 21553 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:11.397814 23449 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.406241 23613 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:11.407332 23626 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250114 08:59:11.408843 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.397977 23617 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:11.369074 23593 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.427631 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1213) took 1424 ms. Trace:
I20250114 08:59:11.375633 23599 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.409832 23588 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:11.438000 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 64376.9us
I20250114 08:59:11.428550 23519 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:11.438733 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.436028 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.439524 21699 rpcz_store.cc:276] 0114 08:59:10.003040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.601806 (+598766us) service_pool.cc:225] Handling call
0114 08:59:11.427395 (+825589us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002e9854. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:11.427565 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282188160}
I20250114 08:59:11.403499 23392 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.386597 23580 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.443128 21704 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
W20250114 08:59:11.443463 21668 leader_election.cc:336] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.473s (SENT)
I20250114 08:59:11.445580 21521 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:11.446141 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 302794us
I20250114 08:59:11.446686 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.447201 21699 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.448330 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:11.449024 21622 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:11.425805 23569 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.449697 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.450501 22274 raft_consensus.cc:1235] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
W20250114 08:59:11.451064 21668 leader_election.cc:336] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.514s (SENT)
I20250114 08:59:11.454020 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:11.454267 21521 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.440062 23617 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.455288 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.456284 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.459240 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.459872 21704 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.461007 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.461226 23617 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:11.461400 23614 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.461645 21627 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.462509 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.462960 21521 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.464097 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.464227 21626 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:11.466373 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.464864 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.438380 23440 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.474231 23588 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.475227 21554 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:11.468899 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:11.469010 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.476903 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.450263 23225 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.478312 21622 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20250114 08:59:11.479964 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.477560 21621 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.439512 23449 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.474876 23440 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.465911 21546 raft_consensus.cc:2405] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:11.483145 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.463305 23613 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.487354 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.468729 23614 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.479614 21695 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
W20250114 08:59:11.492151 21668 leader_election.cc:336] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.565s (SENT)
I20250114 08:59:11.493587 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.487462 23449 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.495571 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.497713 21546 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:11.499711 22270 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:11.499809 23593 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.482024 23225 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:11.503554 23225 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:11.485823 23358 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.504119 23597 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.505318 23597 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:11.500269 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:11.507897 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:11.508725 21546 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:11.509562 23580 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.510744 23609 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.512355 22269 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:11.513123 22269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49004 (request call id 305) took 1689 ms. Trace:
I20250114 08:59:11.515398 23482 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:11.494712 23614 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.475793 23399 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.491420 23613 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:11.516387 21695 raft_consensus.cc:604] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:11.524391 21698 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:59:11.524592 21695 raft_consensus.cc:612] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:11.525772 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:11.526415 21695 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:11.527180 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1208) took 1526 ms. Trace:
I20250114 08:59:11.527626 21546 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:11.527566 21698 rpcz_store.cc:276] 0114 08:59:10.000796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.000939 (+ 143us) service_pool.cc:225] Handling call
0114 08:59:10.001296 (+ 357us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.524806 (+1523510us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.526911 (+ 2105us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540002f154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:11.527091 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3501905664}
I20250114 08:59:11.528571 23596 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.529126 23596 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:11.531562 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 891341us
I20250114 08:59:11.532042 21453 catalog_manager.cc:5526] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 2 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:11.533883 21453 catalog_manager.cc:5526] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:11.523677 23399 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.537942 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:11.538906 21700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1210) took 1536 ms (client timeout 1764 ms). Trace:
I20250114 08:59:11.539271 23399 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.541402 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:11.542342 22239 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:11.537413 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1223) took 1278 ms. Trace:
I20250114 08:59:11.543339 21703 rpcz_store.cc:276] 0114 08:59:10.259241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.569232 (+309991us) service_pool.cc:225] Handling call
0114 08:59:11.537192 (+967960us) spinlock_profiling.cc:243] Waited 9.04 ms on lock 0x7b5400170054. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:11.537346 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20782336}
I20250114 08:59:11.544150 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:11.539307 21700 rpcz_store.cc:269] 0114 08:59:10.002133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.313018 (+310885us) service_pool.cc:225] Handling call
0114 08:59:11.538700 (+1225682us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b5400129154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:11.538819 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2002682752}
W20250114 08:59:11.524024 21452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55888 (request call id 195) took 16466 ms (client timeout 15000 ms). Trace:
I20250114 08:59:11.554533 21518 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:11.550719 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:11.558521 21698 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:11.565953 21594 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:11.570315 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 69067.5us
I20250114 08:59:11.513459 22269 rpcz_store.cc:276] 0114 08:59:09.824042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.826672 (+ 2630us) service_pool.cc:225] Handling call
0114 08:59:11.511930 (+1685258us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:11.512157 (+ 227us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.512855 (+ 698us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.513059 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25091328}
I20250114 08:59:11.571550 21594 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:11.571969 21626 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:11.575022 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.578006 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.579157 21627 raft_consensus.cc:2405] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:11.580811 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.524085 23613 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:11.518939 23614 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:11.585750 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 100796us
W20250114 08:59:11.601359 21668 leader_election.cc:336] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.656s (SENT)
I20250114 08:59:11.601956 21668 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.559885 23449 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:11.601213 21594 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:11.563402 23436 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:11.515678 23437 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.599006 21703 raft_consensus.cc:2388] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:11.562561 23399 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.523358 23599 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.509619 23513 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:11.595575 21698 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:11.647688 21453 catalog_manager.cc:5526] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:11.649861 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.650727 21698 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 1.
I20250114 08:59:11.651407 23569 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:11.651664 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.652031 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:11.612643 23480 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:11.607442 23257 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:11.573875 23385 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:11.652264 21698 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:11.654881 21453 catalog_manager.cc:5526] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:11.655822 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:11.656710 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 47183.6us
I20250114 08:59:11.657413 23444 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.010s
I20250114 08:59:11.661674 21629 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:11.660320 21453 catalog_manager.cc:5526] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 2 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:11.647754 21594 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:11.612341 23051 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:11.663652 23051 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:11.664019 23051 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:11.648102 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 46782.4us
I20250114 08:59:11.663136 21453 catalog_manager.cc:5526] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 2, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 2 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:11.656939 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:11.667501 21668 leader_election.cc:336] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.682s (SENT)
I20250114 08:59:11.667622 21698 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:11.617061 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.585294 23256 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 08:59:11.668332 21668 leader_election.cc:336] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 3.100s (SENT)
I20250114 08:59:11.612128 23429 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20250114 08:59:11.677623 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1232) took 1799 ms. Trace:
I20250114 08:59:11.678117 21624 rpcz_store.cc:276] 0114 08:59:09.877932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.878094 (+ 162us) service_pool.cc:225] Handling call
0114 08:59:11.666677 (+1788583us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54002f9c54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:11.677551 (+ 10874us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235179392}
I20250114 08:59:11.668591 21700 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.582201 23392 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.586162 23443 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:11.688946 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.673540 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:11.678884 21549 raft_consensus.cc:2388] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:11.586472 23547 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:11.680682 21521 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.678611 21698 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:59:11.671200 23256 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:11.585736 23500 consensus_queue.cc:1046] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20250114 08:59:11.694660 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.695046 21521 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.699947 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:11.703182 21698 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:11.704126 21698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
I20250114 08:59:11.690392 21629 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:11.704756 21698 raft_consensus.cc:604] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:11.705206 21698 raft_consensus.cc:612] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:11.666778 23436 raft_consensus.cc:695] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:11.706792 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.710513 21521 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.710635 21624 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:11.711438 21521 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.682682 23385 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:11.665831 23449 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:11.712204 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.712267 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.706904 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:11.744366 21700 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
W20250114 08:59:11.744987 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 76673.1us
I20250114 08:59:11.702596 23256 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:11.712872 23183 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.065s user 0.017s sys 0.025s
I20250114 08:59:11.749945 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.750216 21521 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:11.745160 21703 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:11.687304 23609 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.755570 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.704372 23440 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
W20250114 08:59:11.757655 21668 leader_election.cc:336] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 3.765s (SENT)
I20250114 08:59:11.757880 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1216) took 1738 ms. Trace:
I20250114 08:59:11.760008 21668 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.681842 23399 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.741971 23435 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:11.767781 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.769519 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.714613 23385 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:11.774096 21703 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:11.774736 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1219) took 1754 ms. Trace:
W20250114 08:59:11.756448 23183 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.108s user 0.018s sys 0.026s
I20250114 08:59:11.772796 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.771713 21700 pending_rounds.cc:77] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 1
I20250114 08:59:11.787796 21700 pending_rounds.cc:99] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:11.788100 21700 raft_consensus.cc:2883] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:11.775707 21520 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:11.775573 21703 rpcz_store.cc:276] 0114 08:59:10.019952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.754883 (+1734931us) service_pool.cc:225] Handling call
0114 08:59:11.773787 (+ 18904us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.774477 (+ 690us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.774666 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:11.789418 23449 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.778234 21596 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:11.778016 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.763474 23422 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:11.807255 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.809067 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.769551 23588 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.817162 23422 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.817359 23588 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.817721 23422 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:11.817869 23588 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:11.761296 23635 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.705987 23547 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.854713 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.774223 23435 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.857942 23435 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:11.698734 23443 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.870563 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.871160 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1220) took 1850 ms. Trace:
I20250114 08:59:11.792002 23358 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:11.772550 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:11.778671 23385 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:11.915822 21546 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:11.876518 23599 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:11.918182 21629 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250114 08:59:11.898160 23634 log.cc:926] Time spent T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.133s user 0.000s sys 0.004s
I20250114 08:59:11.887940 21703 rpcz_store.cc:276] 0114 08:59:10.020240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.791336 (+1771096us) service_pool.cc:225] Handling call
0114 08:59:11.870614 (+ 79278us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.870802 (+ 188us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:11.870824 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:11.870865 (+ 41us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:11.870897 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.870952 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:11.871082 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:11.874809 23472 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.808187 23512 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:11.903730 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:11.859692 23635 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:11.875728 23502 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:11.860754 23429 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:11.859381 23474 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:11.561717 21452 rpcz_store.cc:269] 0114 08:58:55.057733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:55.123483 (+ 65750us) service_pool.cc:225] Handling call
0114 08:59:11.523800 (+16400317us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:00.751164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:00.809017 (+ 57853us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:00.809057 (+ 40us) write_op.cc:435] Acquired schema lock
0114 08:59:00.809079 (+ 22us) tablet.cc:659] Decoding operations
0114 08:59:00.810020 (+ 941us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:00.810092 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:00.810133 (+ 41us) tablet.cc:682] Acquiring locks for 6 operations
0114 08:59:00.810848 (+ 715us) tablet.cc:698] Row locks acquired
0114 08:59:00.810869 (+ 21us) write_op.cc:260] PREPARE: finished
0114 08:59:00.811067 (+ 198us) write_op.cc:270] Start()
0114 08:59:00.811205 (+ 138us) write_op.cc:276] Timestamp: P: 1736845140811009 usec, L: 0
0114 08:59:00.811229 (+ 24us) op_driver.cc:348] REPLICATION: starting
0114 08:59:00.911743 (+100514us) log.cc:844] Serialized 2491 byte log entry
0114 08:59:02.493404 (+1581661us) op_driver.cc:464] REPLICATION: finished
0114 08:59:04.145624 (+1652220us) write_op.cc:301] APPLY: starting
0114 08:59:04.145837 (+ 213us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:04.146144 (+ 307us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:04.153380 (+ 7236us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:04.204716 (+ 51336us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:04.205043 (+ 327us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0114 08:59:04.205113 (+ 70us) write_op.cc:312] APPLY: finished
0114 08:59:04.205597 (+ 484us) log.cc:844] Serialized 52 byte log entry
0114 08:59:04.426379 (+220782us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63c67681e 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b
0114 08:59:04.664406 (+238027us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:04.713554 (+ 49148us) write_op.cc:454] Released schema lock
0114 08:59:04.713622 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:04.714100 (+ 478us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:04.714128 (+ 28us) write_op.cc:454] Released schema lock
0114 08:59:04.815434 (+101306us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
Metrics: {"mutex_wait_us":12766,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":1649889,"apply.run_cpu_time_us":24007,"apply.run_wall_time_us":568572,"num_ops":6,"prepare.queue_time_us":659668,"prepare.run_cpu_time_us":38817,"prepare.run_wall_time_us":590459,"raft.queue_time_us":240265,"raft.run_cpu_time_us":68583,"raft.run_wall_time_us":1241514,"replication_time_us":1659333,"spinlock_wait_cycles":97992832,"thread_start_us":2167888,"threads_started":5,"wal-append.queue_time_us":893411,"wal-append.run_cpu_time_us":73795,"wal-append.run_wall_time_us":3174411}]]}
I20250114 08:59:11.849766 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:11.790829 23436 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:11.943140 23399 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.953195 22925 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:11.954416 21550 raft_consensus.cc:2405] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 08:59:11.957126 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 181487us
I20250114 08:59:11.973026 21520 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:11.960144 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 110579us
I20250114 08:59:11.974021 23597 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.974438 21596 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:11.974422 23597 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:11.821038 23617 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.988348 22270 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:11.988615 23437 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:11.994671 23609 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:11.955634 23635 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.031059 21668 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:12.031960 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57646.7us
I20250114 08:59:11.971194 23519 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20250114 08:59:12.052533 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:11.966567 23474 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.997596 23429 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:59:12.038499 23615 consensus_meta.cc:287] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.701s user 0.006s sys 0.026s
I20250114 08:59:11.948239 23472 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:11.957723 23502 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.045867 23609 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:12.034006 23597 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.074344 21549 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:12.075053 21549 raft_consensus.cc:604] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.053833 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:12.076143 21552 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 5
W20250114 08:59:12.031605 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 58533us
I20250114 08:59:12.039201 23467 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:11.998271 23512 consensus_queue.cc:1035] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20250114 08:59:12.077617 21520 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:12.083179 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 240027us
I20250114 08:59:12.085243 21549 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:12.060689 23429 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:12.068380 23609 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.083737 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:12.099082 23400 consensus_meta.cc:287] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.720s user 0.007s sys 0.015s
I20250114 08:59:12.097728 21552 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:12.105237 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.116297 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:12.117208 21552 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:12.118263 23535 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.118263 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:12.118994 23535 consensus_queue.cc:1046] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:59:12.119591 21552 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:12.119886 23535 raft_consensus.cc:3049] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:12.120915 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:12.094491 23429 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:12.067188 23615 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.114635 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:59:12.137151 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:12.138059 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:12.120286 23535 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.093700 23611 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:12.068617 23614 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.104205 23500 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:12.094723 23440 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:12.122280 23482 consensus_meta.cc:287] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.599s user 0.006s sys 0.007s
I20250114 08:59:12.108701 23547 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.126861 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:12.114580 23421 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:12.155130 21552 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250114 08:59:12.087436 23467 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.179123 21553 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 08:59:12.162427 21697 raft_consensus.cc:2405] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:12.121905 23519 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.168197 21520 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:12.169250 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 138313us
I20250114 08:59:12.192998 21668 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.194361 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:12.170198 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 117775us
I20250114 08:59:12.195654 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:12.197674 21596 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:12.144034 23592 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:12.202126 21520 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:12.153513 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:12.154398 23500 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:12.213848 23500 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:12.145156 23537 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250114 08:59:12.216193 23537 consensus_queue.cc:1225] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:12.216746 23537 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:12.192317 23611 raft_consensus.cc:2758] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250114 08:59:12.206238 23480 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.220997 23480 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.210203 23482 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:12.222416 23482 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:12.222980 23482 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:12.210325 23519 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.210772 23399 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 08:59:12.211346 23641 log.cc:926] Time spent T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20250114 08:59:12.135633 23400 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.192512 23256 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.211215 23437 consensus_queue.cc:1046] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:12.211970 22925 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:59:12.217233 23537 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:12.203351 21695 consensus_meta.cc:287] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.676s user 0.013s sys 0.000s
I20250114 08:59:12.216647 23535 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.216853 23592 raft_consensus.cc:695] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:12.209414 21668 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.229000 21555 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:12.228458 21626 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:12.238140 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:12.268155 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 171478us
I20250114 08:59:12.268852 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1217) took 2249 ms. Trace:
W20250114 08:59:12.236150 23613 consensus_meta.cc:287] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.626s user 0.011s sys 0.009s
I20250114 08:59:12.269470 21698 rpcz_store.cc:276] 0114 08:59:10.019356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.750143 (+1730787us) service_pool.cc:225] Handling call
0114 08:59:11.768235 (+ 18092us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:11.768612 (+ 377us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:11.768641 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:11.791135 (+ 22494us) log.cc:844] Serialized 26 byte log entry
0114 08:59:12.098387 (+307252us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:12.267703 (+169316us) log.cc:844] Serialized 14 byte log entry
0114 08:59:12.268509 (+ 806us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.268589 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:12.268632 (+ 43us) raft_consensus.cc:1680] finished
0114 08:59:12.268643 (+ 11us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:12.268798 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21946,"raft.queue_time_us":4875,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":8630,"thread_start_us":321824,"threads_started":2,"wal-append.queue_time_us":362490}
I20250114 08:59:12.224644 22269 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 1 FOLLOWER]: Advancing to term 2
W20250114 08:59:12.287293 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 222421us
I20250114 08:59:12.230926 23535 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.228770 23437 raft_consensus.cc:3049] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:12.289261 21702 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:12.231163 22925 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.230010 23537 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.228834 23400 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.237634 23443 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.230216 21695 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:12.229262 23256 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.268589 23519 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:12.246366 23617 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.268348 23513 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.312320 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:12.304545 23051 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.640s user 0.018s sys 0.005s
I20250114 08:59:12.351267 21698 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:12.159819 23644 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.142s
I20250114 08:59:12.351316 21703 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [5.2-5.2]
W20250114 08:59:12.303696 21580 heartbeater.cc:643] Failed to heartbeat to 127.19.231.190:44737 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.19.231.190:44737 timed out after 15.000s (SENT)
I20250114 08:59:12.307623 23502 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.300165 23474 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.356837 21521 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:12.307961 22271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49020 (request call id 159) took 2468 ms (client timeout 1959 ms). Trace:
I20250114 08:59:12.304419 23437 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.306149 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:12.314234 23519 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:12.288973 23613 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.356175 23472 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:12.358678 23653 log.cc:926] Time spent T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20250114 08:59:12.235842 23592 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.353895 23443 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.354414 23512 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:12.354840 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:12.367267 21521 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.362504 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:12.401036 22273 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:59:12.401820 23435 consensus_meta.cc:287] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.543s user 0.009s sys 0.002s
I20250114 08:59:12.401873 23617 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:12.402755 23617 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.313400 22925 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.403641 23617 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.371325 23502 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.404769 22925 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.404693 23502 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:12.371912 22271 rpcz_store.cc:269] 0114 08:59:09.847019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:09.893314 (+ 46295us) service_pool.cc:225] Handling call
0114 08:59:12.286169 (+2392855us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b540059b0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:12.303662 (+ 17493us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":662132480}
I20250114 08:59:12.406680 23502 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.363346 23661 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:12.375695 23599 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:12.409435 23599 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:12.409662 23613 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.409890 23599 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:12.411681 22269 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:12.358772 23051 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.305915 23440 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.412403 23482 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.413141 23385 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.414176 23051 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.414528 23482 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.378847 23519 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:12.414987 23615 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.371621 23256 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20250114 08:59:12.385208 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.415619 21629 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:12.377379 23500 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.416397 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:12.402366 23435 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:12.426213 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 222555us
W20250114 08:59:12.394414 23648 log.cc:926] Time spent T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20250114 08:59:12.460076 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1218) took 2440 ms. Trace:
I20250114 08:59:12.460515 23500 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.460976 23392 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:12.461532 23392 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.460484 21700 rpcz_store.cc:276] 0114 08:59:10.019647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.751278 (+1731631us) service_pool.cc:225] Handling call
0114 08:59:11.771462 (+ 20184us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:11.858127 (+ 86665us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:11.858162 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:11.858592 (+ 430us) log.cc:844] Serialized 26 byte log entry
0114 08:59:12.069267 (+210675us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:12.390552 (+321285us) log.cc:844] Serialized 14 byte log entry
0114 08:59:12.390853 (+ 301us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.390953 (+ 100us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:12.459869 (+ 68916us) raft_consensus.cc:1680] finished
0114 08:59:12.459887 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:12.460014 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":357625,"raft.queue_time_us":2715,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":7248,"thread_start_us":503962,"threads_started":2,"wal-append.queue_time_us":342529}
I20250114 08:59:12.424474 23614 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.415177 23385 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.454380 21695 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:12.476141 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:12.421398 23512 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:12.465050 23467 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.466596 23482 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.490183 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.491058 21700 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
W20250114 08:59:12.492733 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 61669.1us
W20250114 08:59:12.493443 21668 leader_election.cc:336] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.929s (SENT)
I20250114 08:59:12.493803 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.497642 21594 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:12.497818 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.499974 21594 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:12.504295 21630 raft_consensus.cc:1235] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:12.492926 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:12.486649 23597 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.515057 23467 consensus_queue.cc:1046] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 08:59:12.515946 23467 raft_consensus.cc:3049] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:12.516337 23467 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:12.517277 23467 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.518478 23467 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:12.463220 23437 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.521677 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 145) took 2028 ms. Trace:
I20250114 08:59:12.501510 23644 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:12.522073 21628 rpcz_store.cc:276] 0114 08:59:10.493455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.493946 (+ 491us) service_pool.cc:225] Handling call
0114 08:59:11.666801 (+1172855us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.667015 (+ 214us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:11.667042 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:11.667081 (+ 39us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:11.667113 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.667165 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:12.521482 (+854317us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002f9c50. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:12.521603 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2659198720}
I20250114 08:59:12.526461 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.529222 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.530297 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.530418 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.531059 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.531368 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.496031 23482 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.533990 21624 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:12.516103 23597 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.535768 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:12.533650 23472 consensus_queue.cc:1046] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250114 08:59:12.537946 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.538200 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.538448 23472 raft_consensus.cc:3049] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:12.538805 23472 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:12.539052 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.539247 23482 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.539441 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.540551 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.540123 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.543073 23435 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.539595 23472 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.543269 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.540751 23474 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20250114 08:59:12.544073 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1333) took 1092 ms. Trace:
I20250114 08:59:12.544449 21623 rpcz_store.cc:276] 0114 08:59:11.452124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.543009 (+1090885us) service_pool.cc:225] Handling call
0114 08:59:12.543993 (+ 984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:12.544972 23589 raft_consensus.cc:3049] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:12.522993 23664 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:12.545457 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.545650 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.548767 21700 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:12.549849 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.524238 23644 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:12.552075 22273 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:12.555529 23429 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.557000 21521 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.557653 23429 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.525482 23519 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.558205 23597 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:12.558296 23429 consensus_queue.cc:1046] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:12.558954 23597 raft_consensus.cc:695] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.559561 23644 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.560344 23519 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.559942 23597 consensus_queue.cc:237] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.561038 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.562461 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.564069 21623 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:12.562707 21630 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:12.558207 23537 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.566478 21547 raft_consensus.cc:2363] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:59:12.567358 23537 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.550277 23437 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.568709 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.544087 23472 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.571091 21700 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.547878 23589 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.576088 23589 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.577435 23589 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:12.581454 21627 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250114 08:59:12.574270 23422 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.756s user 0.009s sys 0.004s
I20250114 08:59:12.587388 21623 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:12.591236 21623 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:12.574898 21546 consensus_meta.cc:287] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.658s user 0.007s sys 0.013s
I20250114 08:59:12.588235 22273 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:12.588743 23512 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:12.598361 21623 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.594193 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:12.590233 23635 consensus_meta.cc:287] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.540s user 0.006s sys 0.004s
W20250114 08:59:12.580518 23588 consensus_meta.cc:287] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.762s user 0.013s sys 0.009s
I20250114 08:59:12.603654 23588 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.604770 23599 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.595187 21546 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:12.608412 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:12.608564 23669 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.084s user 0.002s sys 0.000s
I20250114 08:59:12.595763 23422 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.606231 23588 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.610770 23422 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.612879 21549 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:12.605563 23635 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.596838 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.615072 21594 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:12.619297 21520 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.620395 21700 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:12.615104 23537 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:12.688930 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.688763 21521 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.689669 21627 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:12.689394 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 65367.9us
I20250114 08:59:12.689358 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.620788 23385 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:12.691860 21520 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:12.688814 23588 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:12.691228 23615 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.691354 23422 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.707044 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.698298 23435 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:12.709945 21700 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:12.706138 23440 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.711758 23440 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.714563 23440 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.699581 23617 consensus_queue.cc:1046] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
W20250114 08:59:12.700642 22241 leader_election.cc:336] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.596s (SENT)
I20250114 08:59:12.718034 22241 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:12.718935 21700 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:12.719286 21546 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:12.701570 23385 raft_consensus.cc:695] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:12.721987 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.722044 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.721724 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.721606 21521 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.733424 22925 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:12.733633 23617 consensus_queue.cc:1225] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20250114 08:59:12.736177 23609 consensus_meta.cc:287] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.637s user 0.006s sys 0.006s
W20250114 08:59:12.736966 23667 log.cc:926] Time spent T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20250114 08:59:12.720320 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:12.739537 21624 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.769691 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.773787 21700 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:12.773308 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 51781.3us
I20250114 08:59:12.705842 23546 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.784387 21695 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:12.781903 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1353) took 1287 ms. Trace:
I20250114 08:59:12.800257 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.801450 21695 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.784181 23479 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:12.784921 23635 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.785403 23385 consensus_queue.cc:237] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.788731 23599 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.771983 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:12.815621 22271 raft_consensus.cc:2405] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:12.789916 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1263) took 1538 ms. Trace:
I20250114 08:59:12.735296 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:12.779371 23614 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:12.816514 21549 rpcz_store.cc:276] 0114 08:59:11.251708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.086176 (+834468us) service_pool.cc:225] Handling call
0114 08:59:12.612624 (+526448us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:12.613291 (+ 667us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.789660 (+176369us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b540031c750. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:12.789850 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1209721088}
I20250114 08:59:12.779587 23609 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:12.818269 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:12.819660 23609 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:12.819736 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:12.820318 21630 raft_consensus.cc:1235] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:12.821764 21549 raft_consensus.cc:2388] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:12.823005 21521 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.823815 23350 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:12.824440 23350 raft_consensus.cc:695] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.825335 23350 consensus_queue.cc:237] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.827265 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:12.806967 23502 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:12.829085 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1346) took 1582 ms. Trace:
W20250114 08:59:12.794953 21520 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50144 dropped due to backpressure: service queue is full with 50 items
I20250114 08:59:12.829768 23467 consensus_queue.cc:1046] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250114 08:59:12.829452 21553 rpcz_store.cc:276] 0114 08:59:11.246414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.456690 (+210276us) service_pool.cc:225] Handling call
0114 08:59:11.472949 (+ 16259us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:12.192914 (+719965us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.828843 (+635929us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b540024ead4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:12.829022 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1585417600}
I20250114 08:59:12.800843 21625 rpcz_store.cc:276] 0114 08:59:11.494730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.497498 (+ 2768us) service_pool.cc:225] Handling call
0114 08:59:11.570864 (+ 73366us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:11.654036 (+ 83172us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:11.654067 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:11.654498 (+ 431us) log.cc:844] Serialized 26 byte log entry
0114 08:59:11.778662 (+124164us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:12.089377 (+310715us) log.cc:844] Serialized 14 byte log entry
0114 08:59:12.089616 (+ 239us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.781336 (+691720us) spinlock_profiling.cc:243] Waited 82.9 ms on lock 0x7b54001667d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:12.781404 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:12.781487 (+ 83us) raft_consensus.cc:1680] finished
0114 08:59:12.781618 (+ 131us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:12.781825 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":273502,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":73445,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":9085,"spinlock_wait_cycles":190693120,"thread_start_us":270878,"threads_started":2,"wal-append.queue_time_us":287299}
I20250114 08:59:12.765691 23592 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.791157 22269 raft_consensus.cc:1235] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:59:12.794796 23493 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:12.826321 23614 raft_consensus.cc:695] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:12.826321 21696 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1243) took 2189 ms (client timeout 2251 ms). Trace:
W20250114 08:59:12.849220 21696 rpcz_store.cc:269] 0114 08:59:10.637070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.744279 (+107209us) service_pool.cc:225] Handling call
0114 08:59:12.826124 (+2081845us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b540023fad4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:12.826255 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":862117632}
I20250114 08:59:12.831837 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1522) took 1669 ms. Trace:
I20250114 08:59:12.850724 21697 rpcz_store.cc:276] 0114 08:59:11.159512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.258033 (+ 98521us) service_pool.cc:225] Handling call
0114 08:59:12.828294 (+1570261us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b540018a6d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:12.828426 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1956220160}
I20250114 08:59:12.796746 23644 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.808517 21700 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:12.740574 23664 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:12.835955 21627 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:12.766515 23435 raft_consensus.cc:695] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.863106 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.863993 21700 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:12.720801 23615 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.721592 23588 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.852578 21521 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.748955 23392 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.866609 21521 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.867116 23392 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:12.867806 23392 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.860225 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:12.870043 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.837635 23436 raft_consensus.cc:3049] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:12.875736 21695 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:12.870875 21697 raft_consensus.cc:2405] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:12.880200 21521 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.883530 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:12.829952 21520 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50144 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1295)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1281)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1299)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1284)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1294)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1353)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 125)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1354)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 126)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1358)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1363)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1364)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1366)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1381)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1297)
I20250114 08:59:12.887207 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:12.888156 21695 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.888756 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:12.882854 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 215550us
I20250114 08:59:12.889539 21696 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:12.839764 23444 consensus_queue.cc:1046] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 08:59:12.890317 22266 raft_consensus.cc:2405] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250114 08:59:12.891950 21697 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20250114 08:59:12.905071 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1222) took 2711 ms. Trace:
I20250114 08:59:12.905603 21697 rpcz_store.cc:276] 0114 08:59:10.193830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.891346 (+2697516us) service_pool.cc:225] Handling call
0114 08:59:12.891689 (+ 343us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:12.904684 (+ 12995us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.904990 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:12.893905 21521 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.906896 21696 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:12.907642 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:12.853355 21546 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:12.873975 23392 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:12.871467 23436 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:12.909400 21695 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:12.910364 21700 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:12.921362 21700 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:12.922466 21695 raft_consensus.cc:3049] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:12.891474 23437 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:12.916996 21697 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:12.923633 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1225) took 2576 ms. Trace:
I20250114 08:59:12.924064 21697 rpcz_store.cc:276] 0114 08:59:10.347153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.916531 (+2569378us) service_pool.cc:225] Handling call
0114 08:59:12.916830 (+ 299us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.923299 (+ 6469us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.923564 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:12.925591 21697 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:12.917948 21696 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:12.922821 21695 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:12.931576 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1228) took 2428 ms. Trace:
I20250114 08:59:12.917862 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.931946 21696 rpcz_store.cc:276] 0114 08:59:10.502970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.917384 (+2414414us) service_pool.cc:225] Handling call
0114 08:59:12.917730 (+ 346us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.931353 (+ 13623us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.931524 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:12.933073 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.933858 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.916778 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.928335 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1229) took 2422 ms. Trace:
I20250114 08:59:12.934756 21700 rpcz_store.cc:276] 0114 08:59:10.508530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.920928 (+2412398us) service_pool.cc:225] Handling call
0114 08:59:12.921208 (+ 280us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.928033 (+ 6825us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.928263 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:12.886997 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 187538us
I20250114 08:59:12.931769 21695 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:12.938123 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 108194us
I20250114 08:59:12.938794 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1231) took 2398 ms. Trace:
I20250114 08:59:12.910043 23535 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:12.968042 21695 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:12.940017 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:12.969941 21697 rpcz_store.cc:276] 0114 08:59:10.540016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.925002 (+2384986us) service_pool.cc:225] Handling call
0114 08:59:12.925370 (+ 368us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.938526 (+ 13156us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.938727 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:12.939478 21630 raft_consensus.cc:1270] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:12.914927 23440 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:12.971231 21697 raft_consensus.cc:1235] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:12.910451 23670 raft_consensus.cc:3049] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:12.970413 21625 raft_consensus.cc:2405] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:12.892722 23609 consensus_queue.cc:1046] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20250114 08:59:12.918694 23490 consensus_queue.cc:1046] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20250114 08:59:13.004870 21521 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:13.008296 21521 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:12.937825 23635 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:12.997601 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1236) took 2421 ms. Trace:
I20250114 08:59:12.997852 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1235) took 2426 ms. Trace:
I20250114 08:59:12.993672 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:12.929572 23614 consensus_queue.cc:237] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:12.982494 21696 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:13.019572 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1233) took 2449 ms. Trace:
I20250114 08:59:13.019958 21696 rpcz_store.cc:276] 0114 08:59:10.570285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.932700 (+2362415us) service_pool.cc:225] Handling call
0114 08:59:12.982240 (+ 49540us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.019323 (+ 37083us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.019507 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:12.972076 23300 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:13.021378 21700 rpcz_store.cc:276] 0114 08:59:10.571002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.935561 (+2364559us) service_pool.cc:225] Handling call
0114 08:59:12.997309 (+ 61748us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.997495 (+ 186us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:12.997517 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:12.997556 (+ 39us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:12.997591 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.997658 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:12.997784 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.012599 21697 rpcz_store.cc:276] 0114 08:59:10.576424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.970786 (+2394362us) service_pool.cc:225] Handling call
0114 08:59:12.971078 (+ 292us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.997232 (+ 26154us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:12.997524 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.011289 21521 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:12.969275 23392 raft_consensus.cc:695] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:12.999995 23670 raft_consensus.cc:738] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:13.026460 21624 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:12.972656 23437 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.031239 21697 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:13.031272 21696 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250114 08:59:13.011859 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.037693 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1505) took 2312 ms. Trace:
I20250114 08:59:12.937062 23435 consensus_queue.cc:237] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.012686 23400 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:13.035857 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1237) took 2456 ms. Trace:
I20250114 08:59:13.010632 23535 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:13.024770 21695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1230) took 2503 ms. Trace:
I20250114 08:59:13.038333 21697 rpcz_store.cc:276] 0114 08:59:10.725581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.030257 (+2304676us) service_pool.cc:225] Handling call
0114 08:59:13.031084 (+ 827us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.037437 (+ 6353us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.037633 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.023942 23588 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.041862 21696 rpcz_store.cc:276] 0114 08:59:10.579620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.020906 (+2441286us) service_pool.cc:225] Handling call
0114 08:59:13.031072 (+ 10166us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:13.035618 (+ 4546us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.035814 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.054046 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1504) took 2337 ms. Trace:
I20250114 08:59:13.063928 21700 rpcz_store.cc:276] 0114 08:59:10.716702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.030401 (+2313699us) service_pool.cc:225] Handling call
0114 08:59:13.050432 (+ 20031us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.051795 (+ 1363us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.051942 (+ 147us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.052218 (+ 276us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:13.052381 (+ 163us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.053040 (+ 659us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.053847 (+ 807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.068185 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:13.056640 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:13.042203 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 124925us
W20250114 08:59:13.046581 23640 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 1.109s user 0.002s sys 0.000s
I20250114 08:59:13.032366 23350 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:13.061002 22266 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:13.006769 23436 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.057127 21695 rpcz_store.cc:276] 0114 08:59:10.521660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.921398 (+2399738us) service_pool.cc:225] Handling call
0114 08:59:12.921654 (+ 256us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.024412 (+102758us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.024441 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.024492 (+ 51us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:13.024523 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.024578 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.024701 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3959,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":1693}
I20250114 08:59:13.021185 23635 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.087278 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1515) took 2326 ms. Trace:
I20250114 08:59:13.043888 23615 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.043547 23535 raft_consensus.cc:695] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.091780 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1506) took 2361 ms. Trace:
I20250114 08:59:13.093343 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1507) took 2350 ms. Trace:
I20250114 08:59:13.093683 21697 rpcz_store.cc:276] 0114 08:59:10.761115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.085939 (+2324824us) service_pool.cc:225] Handling call
0114 08:59:13.086259 (+ 320us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.086477 (+ 218us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.086513 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.086845 (+ 332us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:13.086881 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.086939 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.087225 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.095654 21700 rpcz_store.cc:276] 0114 08:59:10.730750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.066850 (+2336100us) service_pool.cc:225] Handling call
0114 08:59:13.091450 (+ 24600us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.091580 (+ 130us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.091594 (+ 14us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.091620 (+ 26us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:13.091642 (+ 22us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.091672 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.091737 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.061599 21546 raft_consensus.cc:2371] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4. Re-sending same reply.
I20250114 08:59:13.030599 23300 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.108232 23257 consensus_queue.cc:1046] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:13.095800 21696 rpcz_store.cc:276] 0114 08:59:10.743075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.085068 (+2341993us) service_pool.cc:225] Handling call
0114 08:59:13.092997 (+ 7929us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.093130 (+ 133us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.093144 (+ 14us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.093169 (+ 25us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:13.093189 (+ 20us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.093219 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.093292 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.109480 23300 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:13.110743 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1525) took 1871 ms. Trace:
I20250114 08:59:13.111119 21696 rpcz_store.cc:276] 0114 08:59:11.239163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.109861 (+1870698us) service_pool.cc:225] Handling call
0114 08:59:13.110216 (+ 355us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.110411 (+ 195us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.110438 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.110483 (+ 45us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:13.110520 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.110573 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.110686 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.085083 23644 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.112375 23593 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:13.100276 23635 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.092717 23350 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.113075 23593 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.113708 23350 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.093618 23588 consensus_queue.cc:1046] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20250114 08:59:13.115711 21697 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:13.115934 23588 raft_consensus.cc:3049] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:13.043072 23437 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.116369 23588 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.117359 23588 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.118515 23588 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.106232 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1333) took 1925 ms. Trace:
I20250114 08:59:13.118997 21548 rpcz_store.cc:276] 0114 08:59:11.181170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.181377 (+ 207us) service_pool.cc:225] Handling call
0114 08:59:11.192641 (+ 11264us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.246816 (+ 54175us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.105977 (+1859161us) spinlock_profiling.cc:243] Waited 30.3 ms on lock 0x7b540028dcd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:13.106163 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69653376}
I20250114 08:59:13.120229 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.087019 23400 raft_consensus.cc:695] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.116303 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 105) took 1583 ms. Trace:
I20250114 08:59:13.121389 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 156) took 1344 ms. Trace:
I20250114 08:59:13.121856 21629 rpcz_store.cc:276] 0114 08:59:11.776894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.926774 (+149880us) service_pool.cc:225] Handling call
0114 08:59:11.948971 (+ 22197us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.440078 (+491107us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.121151 (+681073us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b54001522d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:13.121331 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1072725888}
I20250114 08:59:13.122092 23400 consensus_queue.cc:237] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:13.123546 21702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45060 (request call id 103) took 3432 ms (client timeout 2094 ms). Trace:
I20250114 08:59:13.121620 21697 rpcz_store.cc:276] 0114 08:59:11.537922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.115127 (+1577205us) service_pool.cc:225] Handling call
0114 08:59:13.115396 (+ 269us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.116080 (+ 684us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.116253 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.124241 23535 consensus_queue.cc:237] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.122089 21520 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:13.125681 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 107) took 1587 ms. Trace:
W20250114 08:59:13.123955 21702 rpcz_store.cc:269] 0114 08:59:09.691500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.226938 (+1535438us) service_pool.cc:225] Handling call
0114 08:59:13.123381 (+1896443us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54002ea254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.123477 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2100946816}
I20250114 08:59:13.126616 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.130365 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1526) took 1890 ms. Trace:
I20250114 08:59:13.129276 21697 rpcz_store.cc:276] 0114 08:59:11.538581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.124947 (+1586366us) service_pool.cc:225] Handling call
0114 08:59:13.125216 (+ 269us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.125376 (+ 160us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.125399 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.125440 (+ 41us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:13.125473 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.125520 (+ 47us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.125619 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.121419 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.130705 21696 rpcz_store.cc:276] 0114 08:59:11.239483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.112059 (+1872576us) service_pool.cc:225] Handling call
0114 08:59:13.112375 (+ 316us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.130030 (+ 17655us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.130063 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.130107 (+ 44us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:13.130140 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.130207 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.130312 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":85}
I20250114 08:59:13.132104 21697 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:59:13.132762 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1537) took 1571 ms. Trace:
I20250114 08:59:13.133044 21696 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:13.133076 21697 rpcz_store.cc:276] 0114 08:59:11.561471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.131515 (+1570044us) service_pool.cc:225] Handling call
0114 08:59:13.131799 (+ 284us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.132530 (+ 731us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.132714 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.134608 21548 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:13.134104 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1538) took 1566 ms. Trace:
I20250114 08:59:13.135378 21696 rpcz_store.cc:276] 0114 08:59:11.567359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.132615 (+1565256us) service_pool.cc:225] Handling call
0114 08:59:13.132872 (+ 257us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.133874 (+ 1002us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.134051 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.137368 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 113) took 1562 ms. Trace:
I20250114 08:59:13.137948 21696 rpcz_store.cc:276] 0114 08:59:11.574530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.136157 (+1561627us) service_pool.cc:225] Handling call
0114 08:59:13.136807 (+ 650us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.137005 (+ 198us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.137030 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.137064 (+ 34us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:13.137097 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.137167 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.137300 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.140453 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1541) took 1460 ms. Trace:
I20250114 08:59:13.145885 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:13.148612 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.150012 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.126807 23635 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:13.150580 21697 raft_consensus.cc:1270] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250114 08:59:13.150836 21548 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:13.151093 21697 pending_rounds.cc:77] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 2
I20250114 08:59:13.151396 21697 pending_rounds.cc:99] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250114 08:59:13.136732 23690 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:13.151659 21697 raft_consensus.cc:2883] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:13.151788 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.152498 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1542) took 1471 ms. Trace:
I20250114 08:59:13.152989 23472 raft_consensus.cc:3049] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:13.152829 21697 rpcz_store.cc:276] 0114 08:59:11.681339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.149377 (+1468038us) service_pool.cc:225] Handling call
0114 08:59:13.150132 (+ 755us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.152222 (+ 2090us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.152419 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.153736 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.147543 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1524) took 1935 ms. Trace:
I20250114 08:59:13.155195 21700 rpcz_store.cc:276] 0114 08:59:11.212147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.098731 (+1886584us) service_pool.cc:225] Handling call
0114 08:59:13.147187 (+ 48456us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.147324 (+ 137us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.147339 (+ 15us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.147366 (+ 27us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:13.147386 (+ 20us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.147415 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.147495 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.154528 21696 rpcz_store.cc:276] 0114 08:59:11.680160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.139453 (+1459293us) service_pool.cc:225] Handling call
0114 08:59:13.139824 (+ 371us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.140070 (+ 246us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.140105 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.140160 (+ 55us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:13.140192 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.140260 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.140387 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.163254 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:13.163914 21621 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
W20250114 08:59:13.209834 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1212) took 3207 ms (client timeout 1977 ms). Trace:
I20250114 08:59:13.212128 22240 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:13.212456 21701 rpcz_store.cc:269] 0114 08:59:10.002733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.538005 (+535272us) service_pool.cc:225] Handling call
0114 08:59:13.209535 (+2671530us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54002b1454. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.209725 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":475811584}
I20250114 08:59:13.194733 21548 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:13.229342 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 69782.3us
I20250114 08:59:13.157896 23431 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:13.153647 23472 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:13.145834 23467 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20250114 08:59:13.209504 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:13.218521 23683 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:13.095535 23670 consensus_queue.cc:260] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.133816 23392 consensus_queue.cc:237] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.244964 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.231590 23589 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20250114 08:59:13.253309 23490 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:13.251482 23593 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.255745 23683 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:13.288440 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 130959us
I20250114 08:59:13.310719 21520 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:13.249593 23431 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:13.282208 21596 leader_election.cc:336] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.517s (SENT)
W20250114 08:59:13.318492 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 1.72002e+06us
I20250114 08:59:13.278398 21548 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:13.228184 23688 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250114 08:59:13.307291 23670 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:13.329773 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.319207 21546 raft_consensus.cc:2363] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:59:13.312129 23467 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:13.318497 23683 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:13.288408 23635 raft_consensus.cc:695] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.279690 23490 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.340812 21629 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:59:13.339928 21668 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.345124 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.345996 21548 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.377101 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:13.378463 21696 raft_consensus.cc:2388] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:13.274812 23472 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.324021 23440 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20250114 08:59:13.386710 23635 consensus_queue.cc:237] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.385504 23487 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.387588 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:13.387192 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:13.415319 21630 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:13.415609 21696 raft_consensus.cc:1270] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:59:13.418155 21548 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:13.418610 23596 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:13.419250 23596 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:13.419358 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.419725 23596 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.420298 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1284) took 1165 ms. Trace:
I20250114 08:59:13.420727 21548 rpcz_store.cc:276] 0114 08:59:12.255085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.419102 (+1164017us) service_pool.cc:225] Handling call
0114 08:59:13.420242 (+ 1140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.421855 23672 raft_consensus.cc:3049] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Stepping down as leader of term 1
I20250114 08:59:13.422392 23672 raft_consensus.cc:738] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:13.342355 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1283) took 1085 ms. Trace:
I20250114 08:59:13.343062 23449 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:13.428992 23502 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:13.433070 23597 raft_consensus.cc:988] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:59:13.433568 23597 raft_consensus.cc:1076] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Signalling peer d7732324fb5f48d3b16a29d4bb2584ed to start an election
I20250114 08:59:13.428594 23672 consensus_queue.cc:260] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.429607 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:13.437121 23672 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:13.437083 23400 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:13.438655 23472 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:13.429580 23490 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:13.441926 21547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1277) took 1192 ms (client timeout 1532 ms). Trace:
I20250114 08:59:13.442104 23490 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:13.442341 21547 rpcz_store.cc:269] 0114 08:59:12.249841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.522890 (+273049us) service_pool.cc:225] Handling call
0114 08:59:13.441710 (+918820us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b540007d5d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.441850 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68714752}
I20250114 08:59:13.442596 23490 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.443539 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.444103 21547 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.423020 23431 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:13.341012 23467 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.452221 21594 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:13.453321 23467 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:13.453462 21594 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:13.456005 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:13.459689 23449 raft_consensus.cc:695] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:13.460932 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.461076 23449 consensus_queue.cc:237] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.461109 23431 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:13.389007 21625 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:13.462332 21520 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:13.462554 21625 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:13.431080 23596 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:13.467525 21622 raft_consensus.cc:3049] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:13.467895 21622 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:13.472734 21622 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.473500 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 110) took 1930 ms. Trace:
I20250114 08:59:13.474165 21622 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.463390 21625 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.475837 21625 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.485209 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:13.490708 21697 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:13.491731 21629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:57318
I20250114 08:59:13.492378 21629 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:13.493014 21629 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.492761 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:13.493818 21520 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.500615 21699 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:13.467795 23575 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:13.502314 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1215) took 3483 ms. Trace:
I20250114 08:59:13.502306 23575 raft_consensus.cc:695] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:13.502732 21699 rpcz_store.cc:276] 0114 08:59:10.018744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.491746 (+3473002us) service_pool.cc:225] Handling call
0114 08:59:13.492079 (+ 333us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.502005 (+ 9926us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.502248 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.503526 23575 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.505780 23422 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:13.500247 23596 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20250114 08:59:13.506431 23422 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.509093 22885 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:13.508980 23596 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:13.509573 22885 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:13.510033 22885 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:13.460928 23437 consensus_queue.cc:1046] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250114 08:59:13.513320 23437 consensus_queue.cc:1225] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:13.520440 21622 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:13.521821 21547 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:13.522485 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1379) took 1015 ms. Trace:
I20250114 08:59:13.522842 21547 rpcz_store.cc:276] 0114 08:59:12.507261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.443271 (+936010us) service_pool.cc:225] Handling call
0114 08:59:13.522426 (+ 79155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.525441 21625 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:13.530951 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.519282 21629 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.477120 23597 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:13.426930 21546 rpcz_store.cc:276] 0114 08:59:12.254807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.154706 (+899899us) service_pool.cc:225] Handling call
0114 08:59:13.339691 (+184985us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.540117 21701 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:13.537035 21629 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.542537 21697 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:13.499730 23490 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:13.474074 21702 rpcz_store.cc:276] 0114 08:59:11.543051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.130448 (+1587397us) service_pool.cc:225] Handling call
0114 08:59:13.130760 (+ 312us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:13.131107 (+ 347us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:13.131133 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:13.153376 (+ 22243us) log.cc:844] Serialized 26 byte log entry
0114 08:59:13.336564 (+183188us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:13.343430 (+ 6866us) log.cc:844] Serialized 14 byte log entry
0114 08:59:13.473102 (+129672us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.473208 (+ 106us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:13.473248 (+ 40us) raft_consensus.cc:1680] finished
0114 08:59:13.473266 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.473425 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10651,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":88189,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":11107,"thread_start_us":311800,"threads_started":2,"wal-append.queue_time_us":196302}
I20250114 08:59:13.421824 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.549317 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.553864 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1278) took 1300 ms. Trace:
I20250114 08:59:13.556936 21548 rpcz_store.cc:276] 0114 08:59:12.253414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.421567 (+1168153us) service_pool.cc:225] Handling call
0114 08:59:13.553780 (+132213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.558184 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.558952 22267 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250114 08:59:13.559052 21548 raft_consensus.cc:2388] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 2: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:13.559139 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:13.561174 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.561957 23480 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:13.562371 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.563026 21547 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:13.534103 23615 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:13.544747 23400 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:13.563561 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1294) took 1142 ms. Trace:
I20250114 08:59:13.602560 21547 rpcz_store.cc:276] 0114 08:59:12.421344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.562148 (+1140804us) service_pool.cc:225] Handling call
0114 08:59:13.563498 (+ 1350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.603699 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.560200 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.556118 23589 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250114 08:59:13.609077 22267 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:13.609393 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1285) took 1354 ms. Trace:
I20250114 08:59:13.609809 21548 rpcz_store.cc:276] 0114 08:59:12.255349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.559942 (+1304593us) service_pool.cc:225] Handling call
0114 08:59:13.609332 (+ 49390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.550964 23688 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:13.538895 22241 leader_election.cc:336] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.094s (SENT)
I20250114 08:59:13.614960 21547 raft_consensus.cc:2363] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 6.
I20250114 08:59:13.615947 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.623564 21548 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:13.624214 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1296) took 1083 ms. Trace:
I20250114 08:59:13.624624 21548 rpcz_store.cc:276] 0114 08:59:12.541126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.610469 (+1069343us) service_pool.cc:225] Handling call
0114 08:59:13.624151 (+ 13682us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:13.616679 21554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1256) took 2418 ms (client timeout 1824 ms). Trace:
W20250114 08:59:13.625622 21554 rpcz_store.cc:269] 0114 08:59:11.198364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.200439 (+ 2075us) service_pool.cc:225] Handling call
0114 08:59:13.616484 (+2416045us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b5400323f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.616611 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582568832}
I20250114 08:59:13.618324 23433 consensus_queue.cc:1046] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:59:13.627051 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.632789 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.633514 21548 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:13.634766 23422 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:13.636004 22274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48992 (request call id 208) took 2852 ms. Trace:
I20250114 08:59:13.639029 22239 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:13.648733 21551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1250) took 2452 ms (client timeout 1929 ms). Trace:
I20250114 08:59:13.648975 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:13.649147 21551 rpcz_store.cc:269] 0114 08:59:11.196224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.196574 (+ 350us) service_pool.cc:225] Handling call
0114 08:59:13.648514 (+2451940us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54001176d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.648640 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228761472}
I20250114 08:59:13.649574 21702 raft_consensus.cc:1808] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:13.650090 21702 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:13.555949 23392 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20250114 08:59:13.647274 22274 rpcz_store.cc:276] 0114 08:59:10.783430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:10.783594 (+ 164us) service_pool.cc:225] Handling call
0114 08:59:10.783894 (+ 300us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.457802 (+673908us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.635761 (+2177959us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54006d06d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:13.635940 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532608256}
I20250114 08:59:13.652504 21695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1523) took 2492 ms. Trace:
I20250114 08:59:13.660395 21695 rpcz_store.cc:276] 0114 08:59:11.159769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.094647 (+1934878us) service_pool.cc:225] Handling call
0114 08:59:13.094913 (+ 266us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.095060 (+ 147us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:13.269285 (+174225us) log.cc:844] Serialized 14 byte log entry
0114 08:59:13.652107 (+382822us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:13.652171 (+ 64us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:13.652232 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.652297 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:13.652416 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":188746,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":97641,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"thread_start_us":553741,"threads_started":3,"wal-append.queue_time_us":231281}
I20250114 08:59:13.550519 23683 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:13.652560 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 171193us
W20250114 08:59:13.652624 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 312667us
I20250114 08:59:13.645449 21554 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:13.674211 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1275) took 2420 ms. Trace:
I20250114 08:59:13.674702 21554 rpcz_store.cc:276] 0114 08:59:11.254060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.638389 (+2384329us) service_pool.cc:225] Handling call
0114 08:59:13.638677 (+ 288us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.673835 (+ 35158us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.674127 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.580121 23435 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20250114 08:59:13.614181 23616 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20250114 08:59:13.637827 23688 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:13.678205 21700 raft_consensus.cc:1270] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:13.818692 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:13.777388 21630 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:13.828846 21625 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:13.678113 21668 leader_election.cc:336] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 2.210s (SENT)
W20250114 08:59:13.830263 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 152178us
W20250114 08:59:13.831816 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 99348.2us
I20250114 08:59:13.831962 21668 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.833456 23716 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:13.833827 21668 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:13.834892 23713 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:13.835282 23713 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:13.835606 23713 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.835839 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:13.837579 23710 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:13.836999 21668 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:13.840003 23709 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:13.840628 23709 raft_consensus.cc:695] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:13.841264 21668 leader_election.cc:400] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:59:13.841682 23709 consensus_queue.cc:237] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.839563 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
W20250114 08:59:13.763504 23535 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.028s sys 0.009s
I20250114 08:59:13.825686 23683 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:13.646032 22241 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:13.846354 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 308419us
I20250114 08:59:13.847790 23683 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:13.848608 23718 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:13.849109 23718 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:13.845261 23535 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.031s sys 0.015s
I20250114 08:59:13.801651 22266 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:11.760061 21704 rpcz_store.cc:276] 0114 08:59:10.019065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.468202 (+1449137us) service_pool.cc:225] Handling call
0114 08:59:11.482496 (+ 14294us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:11.482701 (+ 205us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:11.497059 (+ 14358us) log.cc:844] Serialized 14 byte log entry
0114 08:59:11.757542 (+260483us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:11.757591 (+ 49us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:11.757638 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:11.757705 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:11.757823 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2500,"prepare.queue_time_us":15095,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":3974,"thread_start_us":257582,"threads_started":1}
W20250114 08:59:13.738854 23707 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20250114 08:59:13.653856 23467 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:13.738462 23421 raft_consensus.cc:3049] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:13.793092 23711 consensus_queue.cc:1046] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20250114 08:59:13.763828 23433 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:13.855296 23502 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:13.700646 23617 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.161s
I20250114 08:59:13.859756 23472 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:13.867038 23472 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:13.882565 23652 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 1.573s user 0.002s sys 0.000s
I20250114 08:59:13.885650 23718 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:13.893229 23688 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:13.894184 23703 log.cc:926] Time spent T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.604s user 0.008s sys 0.000s
I20250114 08:59:13.900336 23710 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:13.901320 23692 log.cc:926] Time spent T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.610s user 0.003s sys 0.000s
I20250114 08:59:13.901571 23713 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:13.903262 23697 log.cc:926] Time spent T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.621s user 0.000s sys 0.002s
I20250114 08:59:13.903999 23713 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:13.901405 23693 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.614s user 0.002s sys 0.000s
I20250114 08:59:13.936138 21554 raft_consensus.cc:1235] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:13.938457 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1353) took 2612 ms. Trace:
I20250114 08:59:13.939105 21554 rpcz_store.cc:276] 0114 08:59:11.326148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.675631 (+2349483us) service_pool.cc:225] Handling call
0114 08:59:13.676054 (+ 423us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.937628 (+261574us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.938254 (+ 626us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.937376 21451 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 6 to 7, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 7 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250114 08:59:13.941310 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 422650us
I20250114 08:59:13.942757 21554 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:13.943253 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:13.943842 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 125) took 2606 ms. Trace:
I20250114 08:59:13.944525 21554 rpcz_store.cc:276] 0114 08:59:11.337321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.941527 (+2604206us) service_pool.cc:225] Handling call
0114 08:59:13.942386 (+ 859us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:13.943405 (+ 1019us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.943698 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:13.945984 21550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1270) took 2692 ms (client timeout 2210 ms). Trace:
W20250114 08:59:13.946537 21550 rpcz_store.cc:269] 0114 08:59:11.253217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.357566 (+104349us) service_pool.cc:225] Handling call
0114 08:59:13.945408 (+2587842us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b540016f654. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.945729 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1368696704}
I20250114 08:59:13.951692 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1287) took 1695 ms. Trace:
I20250114 08:59:13.956493 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:13.957881 21553 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:13.961580 21594 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:13.963371 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:13.968602 21621 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.014s sys 0.022s
W20250114 08:59:13.969174 21621 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s user 0.014s sys 0.022s
I20250114 08:59:13.974906 21668 leader_election.cc:400] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5.
I20250114 08:59:13.981007 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:13.981612 21626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43436 (request call id 154) took 2322 ms (client timeout 2735 ms). Trace:
I20250114 08:59:13.982350 21624 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:13.978281 21668 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:13.982438 21698 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1544) took 2034 ms (client timeout 2142 ms). Trace:
W20250114 08:59:13.984902 21698 rpcz_store.cc:269] 0114 08:59:11.947814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.362597 (+414783us) service_pool.cc:225] Handling call
0114 08:59:13.980396 (+1617799us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b5800297398. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:13.982293 (+ 1897us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12002688}
I20250114 08:59:13.978415 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1342) took 2754 ms. Trace:
I20250114 08:59:13.991977 21555 rpcz_store.cc:276] 0114 08:59:11.224213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.395261 (+171048us) service_pool.cc:225] Handling call
0114 08:59:11.395478 (+ 217us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:12.239362 (+843884us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.977755 (+1738393us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b5400193f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:13.978200 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1916530560}
I20250114 08:59:13.998658 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:13.969035 21549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 130) took 1567 ms (client timeout 1676 ms). Trace:
I20250114 08:59:13.985177 21668 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:14.003283 21549 rpcz_store.cc:269] 0114 08:59:12.401342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.942442 (+1541100us) service_pool.cc:225] Handling call
0114 08:59:13.968909 (+ 26467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:13.952251 21547 rpcz_store.cc:276] 0114 08:59:12.255915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.603386 (+1347471us) service_pool.cc:225] Handling call
0114 08:59:13.951190 (+347804us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b540028e1d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.951472 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23952640}
I20250114 08:59:13.998706 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1221) took 3937 ms. Trace:
I20250114 08:59:14.024849 21703 rpcz_store.cc:276] 0114 08:59:10.061600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.939498 (+1877898us) service_pool.cc:225] Handling call
0114 08:59:12.314363 (+374865us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:12.358648 (+ 44285us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:13.998207 (+1639559us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400328050. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.998485 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":803826560}
I20250114 08:59:14.027407 23644 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:14.028327 23644 raft_consensus.cc:695] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.029968 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:14.030113 23644 consensus_queue.cc:237] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:13.870899 23472 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:13.859261 23421 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:13.982246 21626 rpcz_store.cc:269] 0114 08:59:11.658804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:11.668688 (+ 9884us) service_pool.cc:225] Handling call
0114 08:59:13.981091 (+2312403us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54000c8d54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:13.981369 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1206828416}
I20250114 08:59:14.032927 23658 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:14.039067 23467 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:13.862205 23433 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:14.049393 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 86300.2us
I20250114 08:59:14.049496 23467 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:14.051573 23729 log.cc:926] Time spent T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20250114 08:59:13.869390 23502 raft_consensus.cc:3049] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:14.052978 23502 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:14.054553 23502 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.055507 23436 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:14.058113 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 73162.4us
I20250114 08:59:14.059304 21668 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:14.061036 23713 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:14.061338 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 126) took 2625 ms. Trace:
I20250114 08:59:14.061575 23713 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.062053 23713 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:14.061821 21550 rpcz_store.cc:276] 0114 08:59:11.435910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.059861 (+2623951us) service_pool.cc:225] Handling call
0114 08:59:14.060458 (+ 597us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.060928 (+ 470us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:14.060968 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:14.061042 (+ 74us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:14.061077 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.061142 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:14.061249 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.063515 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.065805 21555 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:14.066594 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1358) took 2616 ms. Trace:
I20250114 08:59:14.066929 21555 rpcz_store.cc:276] 0114 08:59:11.450108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.065019 (+2614911us) service_pool.cc:225] Handling call
0114 08:59:14.065531 (+ 512us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.066334 (+ 803us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.066534 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.067524 23718 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:14.039685 23683 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20250114 08:59:14.077369 21621 raft_consensus.cc:3049] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:14.077940 21621 raft_consensus.cc:738] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.077791 23510 consensus_queue.cc:1046] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:14.084002 22925 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.084625 22925 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:14.039830 21554 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:14.089578 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1354) took 2749 ms. Trace:
I20250114 08:59:14.089334 23421 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.090122 21554 rpcz_store.cc:276] 0114 08:59:11.340507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.038762 (+2698255us) service_pool.cc:225] Handling call
0114 08:59:14.039486 (+ 724us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.089276 (+ 49790us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.089488 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.090821 23225 raft_consensus.cc:3049] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:14.043624 21630 raft_consensus.cc:3049] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:14.091917 21630 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.092666 21548 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:14.043936 23709 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250114 08:59:14.094203 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.100996 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.042876 21628 raft_consensus.cc:3049] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.104446 21628 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:14.105576 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 46417.3us
I20250114 08:59:14.105665 21628 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.106578 21668 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:14.107018 21628 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.113574 21547 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250114 08:59:14.115839 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.116956 23433 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.091462 23225 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.068172 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:14.121881 23502 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:14.122722 21623 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:14.123597 21546 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:59:14.123662 21624 raft_consensus.cc:3049] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.071966 23683 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20250114 08:59:14.124722 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1362) took 2662 ms. Trace:
I20250114 08:59:14.163599 21547 rpcz_store.cc:276] 0114 08:59:11.462038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.102405 (+2640367us) service_pool.cc:225] Handling call
0114 08:59:14.113080 (+ 10675us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:14.124342 (+ 11262us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.124616 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.091442 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.130757 21555 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:14.149426 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 110039us
W20250114 08:59:14.149320 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 111093us
I20250114 08:59:14.126818 21549 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:14.158398 21624 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.093148 21630 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.091490 23421 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:14.128382 23225 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.175035 23225 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:14.136737 21451 catalog_manager.cc:5526] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:14.177170 21520 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:14.138887 23400 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:14.178090 23400 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.178237 21451 catalog_manager.cc:5526] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 4 to 5, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:14.179363 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1365) took 2686 ms. Trace:
I20250114 08:59:14.179811 21549 rpcz_store.cc:276] 0114 08:59:11.493290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.178421 (+2685131us) service_pool.cc:225] Handling call
0114 08:59:14.178811 (+ 390us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.178991 (+ 180us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:14.179014 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:14.179070 (+ 56us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:14.179100 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.179152 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:14.179273 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.180954 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1361) took 2719 ms. Trace:
I20250114 08:59:14.181385 21546 rpcz_store.cc:276] 0114 08:59:11.461742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.122512 (+2660770us) service_pool.cc:225] Handling call
0114 08:59:14.123030 (+ 518us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.158270 (+ 35240us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.180870 (+ 22600us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.182238 23589 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:14.183084 21546 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:14.183771 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1368) took 2677 ms. Trace:
I20250114 08:59:14.186396 21548 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:14.133702 23467 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.132656 23510 consensus_queue.cc:1225] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250114 08:59:14.187146 21554 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:14.188642 21554 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:14.189224 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1369) took 2683 ms. Trace:
I20250114 08:59:14.140849 23718 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:14.190207 23718 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.190848 21630 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:14.181522 21549 raft_consensus.cc:1270] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 08:59:14.184113 21546 rpcz_store.cc:276] 0114 08:59:11.505911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.182327 (+2676416us) service_pool.cc:225] Handling call
0114 08:59:14.182807 (+ 480us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.183532 (+ 725us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.183721 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:14.146947 22885 consensus_meta.cc:287] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.636s user 0.004s sys 0.011s
I20250114 08:59:14.135485 22266 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:14.189550 21554 rpcz_store.cc:276] 0114 08:59:11.506219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.187962 (+2681743us) service_pool.cc:225] Handling call
0114 08:59:14.188428 (+ 466us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.188991 (+ 563us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.189165 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.171875 21553 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
W20250114 08:59:14.199318 21553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1295) took 1772 ms (client timeout 1667 ms). Trace:
W20250114 08:59:14.199797 21553 rpcz_store.cc:269] 0114 08:59:12.422121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.955839 (+1533718us) service_pool.cc:225] Handling call
0114 08:59:14.194465 (+238626us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.199151 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1367) took 2687 ms. Trace:
I20250114 08:59:14.139322 23449 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.203140 21549 rpcz_store.cc:276] 0114 08:59:11.505566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.180743 (+2675177us) service_pool.cc:225] Handling call
0114 08:59:14.181103 (+ 360us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.192767 (+ 11664us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.193135 (+ 368us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.209179 21554 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20250114 08:59:14.212231 23392 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.213116 23392 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.215325 21549 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:14.215971 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1374) took 2114 ms. Trace:
I20250114 08:59:14.216269 21549 rpcz_store.cc:276] 0114 08:59:12.102087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.214625 (+2112538us) service_pool.cc:225] Handling call
0114 08:59:14.215055 (+ 430us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.215727 (+ 672us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.215900 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.217677 21549 raft_consensus.cc:1235] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:14.218211 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1286) took 1962 ms. Trace:
I20250114 08:59:14.218499 21549 rpcz_store.cc:276] 0114 08:59:12.255618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.217100 (+1961482us) service_pool.cc:225] Handling call
0114 08:59:14.217478 (+ 378us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.218009 (+ 531us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.218163 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.212302 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1373) took 2547 ms. Trace:
I20250114 08:59:14.219854 21554 rpcz_store.cc:276] 0114 08:59:11.666109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.205266 (+2539157us) service_pool.cc:225] Handling call
0114 08:59:14.205725 (+ 459us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:14.212015 (+ 6290us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.212243 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.198716 23683 consensus_queue.cc:1225] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250114 08:59:14.198063 23718 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:14.203122 23400 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.236794 21624 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:14.238866 23614 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.014s sys 0.002s
I20250114 08:59:14.239787 23724 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.241724 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 129) took 2564 ms. Trace:
I20250114 08:59:14.245227 21554 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 08:59:14.246421 23449 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.248243 21555 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:14.251473 21453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 198) took 18135 ms (client timeout 15000 ms). Trace:
I20250114 08:59:14.189885 23599 consensus_queue.cc:1035] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 08:59:14.291523 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1289) took 2035 ms. Trace:
I20250114 08:59:14.386713 21623 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250114 08:59:14.269048 22274 raft_consensus.cc:3049] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Stepping down as leader of term 6
W20250114 08:59:14.291754 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 149517us
I20250114 08:59:14.290678 21630 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 08:59:14.387203 21624 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.401782 21628 raft_consensus.cc:1270] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:14.323732 21547 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 08:59:14.419183 22274 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:14.418228 21554 rpcz_store.cc:276] 0114 08:59:12.256674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.231986 (+1975312us) service_pool.cc:225] Handling call
0114 08:59:14.244981 (+ 12995us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:14.270119 (+ 25138us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.291437 (+ 21318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.259806 21553 rpcz_store.cc:276] 0114 08:59:11.676915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.209952 (+2533037us) service_pool.cc:225] Handling call
0114 08:59:14.241008 (+ 31056us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.241338 (+ 330us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:14.241362 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:14.241424 (+ 62us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:14.241477 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.241537 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:14.241657 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:14.404945 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 327858us
I20250114 08:59:14.423739 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1364) took 2938 ms. Trace:
I20250114 08:59:14.424383 21555 rpcz_store.cc:276] 0114 08:59:11.480913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.171222 (+2690309us) service_pool.cc:225] Handling call
0114 08:59:14.248052 (+ 76830us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.418992 (+170940us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.419183 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.425339 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1363) took 2958 ms. Trace:
I20250114 08:59:14.425843 21547 rpcz_store.cc:276] 0114 08:59:11.462333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.197577 (+2735244us) service_pool.cc:225] Handling call
0114 08:59:14.306964 (+109387us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:14.420502 (+113538us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.420655 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.428020 21546 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250114 08:59:14.295416 23614 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s user 0.015s sys 0.002s
I20250114 08:59:14.398016 21621 consensus_queue.cc:260] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.429198 21549 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:14.435655 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1383) took 1850 ms. Trace:
I20250114 08:59:14.269762 21451 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:14.432114 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1372) took 2767 ms. Trace:
I20250114 08:59:14.439765 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1288) took 2183 ms. Trace:
I20250114 08:59:14.439814 21546 rpcz_store.cc:276] 0114 08:59:11.664569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.199742 (+2535173us) service_pool.cc:225] Handling call
0114 08:59:14.427647 (+227905us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.428392 (+ 745us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.432057 (+ 3665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.440256 21549 rpcz_store.cc:276] 0114 08:59:12.256235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.219242 (+1963007us) service_pool.cc:225] Handling call
0114 08:59:14.417638 (+198396us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.435338 (+ 17700us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.439707 (+ 4369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.436302 21623 rpcz_store.cc:276] 0114 08:59:12.578802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.579532 (+ 730us) service_pool.cc:225] Handling call
0114 08:59:12.587243 (+ 7711us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.121778 (+1534535us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b580042fbb0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5491c3 00007fa63f5acaba 00007fa63f5c43af 00007fa63f5c5b4e 00007fa63f5cfc89 00007fa63f5d0556 00007fa63f5ccedf 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694
0114 08:59:14.418696 (+296918us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.429220 (+ 10524us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3393942272}
W20250114 08:59:14.447402 23298 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250114 08:59:14.339785 23400 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.204038 22885 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.390798 23480 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.453755 21547 raft_consensus.cc:1235] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 1. Current term is 2. Ops: []
I20250114 08:59:14.467711 22274 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.458791 23431 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.459122 23436 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:14.459443 23596 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.461114 23467 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.461936 21554 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:14.465874 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:14.466346 21451 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:14.469587 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1383) took 1634 ms. Trace:
I20250114 08:59:14.334199 23724 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:14.479581 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 62308.7us
I20250114 08:59:14.487980 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1378) took 1981 ms. Trace:
I20250114 08:59:14.481173 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:14.477561 21625 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:14.485432 21547 rpcz_store.cc:276] 0114 08:59:12.835672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.448169 (+1612497us) service_pool.cc:225] Handling call
0114 08:59:14.453489 (+ 5320us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.469339 (+ 15850us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.469534 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.488624 21554 rpcz_store.cc:276] 0114 08:59:12.506925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.426758 (+1919833us) service_pool.cc:225] Handling call
0114 08:59:14.461684 (+ 34926us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.487542 (+ 25858us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.487900 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.483645 21521 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:14.495880 21553 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:14.487920 22267 raft_consensus.cc:1808] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:14.496920 21552 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:14.497398 22267 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:14.500208 21621 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:14.501904 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1381) took 1994 ms. Trace:
I20250114 08:59:14.502415 21553 rpcz_store.cc:276] 0114 08:59:12.507758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.426782 (+1919024us) service_pool.cc:225] Handling call
0114 08:59:14.482595 (+ 55813us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.496455 (+ 13860us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.499182 (+ 2727us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.513214 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:14.484913 23431 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.515508 22274 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:14.519481 23599 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 08:59:14.519851 21551 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
W20250114 08:59:14.520923 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 93897.1us
I20250114 08:59:14.521267 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1351) took 3242 ms. Trace:
I20250114 08:59:14.521620 21551 rpcz_store.cc:276] 0114 08:59:11.279205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:13.649986 (+2370781us) service_pool.cc:225] Handling call
0114 08:59:13.948435 (+298449us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.520289 (+571854us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.521021 (+ 732us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54003a2ad4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:14.521195 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1279620480}
I20250114 08:59:14.522823 23596 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.498059 23480 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.525141 23596 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:14.490835 23467 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.501197 21451 catalog_manager.cc:5526] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 5, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 5 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:14.519014 23709 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:14.494446 23400 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:14.546510 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet b3650a91ff4a4213b9091f197bf75fb9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d7732324fb5f48d3b16a29d4bb2584ed is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:14.549331 21668 leader_election.cc:400] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 6.
I20250114 08:59:14.553352 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1274) took 3299 ms. Trace:
I20250114 08:59:14.554432 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:14.554756 21549 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:59:14.555150 21547 raft_consensus.cc:3049] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:14.555469 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 133) took 1457 ms. Trace:
I20250114 08:59:14.555876 21549 rpcz_store.cc:276] 0114 08:59:13.098197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.474765 (+1376568us) service_pool.cc:225] Handling call
0114 08:59:14.554376 (+ 79611us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.555213 (+ 837us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.555390 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48512}
I20250114 08:59:14.555496 21547 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:14.554770 21552 rpcz_store.cc:276] 0114 08:59:11.253907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:12.164195 (+910288us) service_pool.cc:225] Handling call
0114 08:59:13.945694 (+1781499us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.497460 (+551766us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.527555 (+ 30095us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b540016f654. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:14.550729 (+ 23174us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b540016f650. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:14.553299 (+ 2570us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5340262016}
I20250114 08:59:14.548002 23589 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:14.563357 21551 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250114 08:59:14.555744 21628 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:14.569365 21554 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
W20250114 08:59:14.561200 23472 consensus_meta.cc:287] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.508s user 0.007s sys 0.006s
I20250114 08:59:14.565172 21553 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250114 08:59:14.569067 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 642321us
I20250114 08:59:14.573426 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1303) took 1235 ms. Trace:
I20250114 08:59:14.573968 22240 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:14.578186 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1385) took 1107 ms. Trace:
I20250114 08:59:14.578673 21553 rpcz_store.cc:276] 0114 08:59:13.470855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.512703 (+1041848us) service_pool.cc:225] Handling call
0114 08:59:14.557821 (+ 45118us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.577841 (+ 20020us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.578115 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21888}
I20250114 08:59:14.578802 21547 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.579434 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:14.580080 21547 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:14.580672 23467 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.580858 22925 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.581414 23467 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:14.581979 23467 raft_consensus.cc:695] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:14.583235 22925 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.582988 23467 consensus_queue.cc:237] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.583842 23683 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:14.558609 23614 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20250114 08:59:14.584304 23683 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.584707 23683 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:14.420804 21453 rpcz_store.cc:269] 0114 08:58:56.116299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:56.116573 (+ 274us) service_pool.cc:225] Handling call
0114 08:59:14.251098 (+18134525us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:12.217897 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:12.218522 (+ 625us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:12.218576 (+ 54us) write_op.cc:435] Acquired schema lock
0114 08:59:12.218595 (+ 19us) tablet.cc:659] Decoding operations
0114 08:59:12.219432 (+ 837us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:12.219523 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:12.219553 (+ 30us) tablet.cc:682] Acquiring locks for 7 operations
0114 08:59:12.220240 (+ 687us) tablet.cc:698] Row locks acquired
0114 08:59:12.220259 (+ 19us) write_op.cc:260] PREPARE: finished
0114 08:59:12.220425 (+ 166us) write_op.cc:270] Start()
0114 08:59:12.235173 (+ 14748us) write_op.cc:276] Timestamp: P: 1736845152220362 usec, L: 0
0114 08:59:12.235196 (+ 23us) op_driver.cc:348] REPLICATION: starting
0114 08:59:12.235865 (+ 669us) log.cc:844] Serialized 2890 byte log entry
0114 08:59:12.712136 (+476271us) op_driver.cc:464] REPLICATION: finished
0114 08:59:12.857360 (+145224us) write_op.cc:301] APPLY: starting
0114 08:59:12.861172 (+ 3812us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:12.868686 (+ 7514us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:12.868832 (+ 146us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:13.027702 (+158870us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:13.060847 (+ 33145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0114 08:59:13.063604 (+ 2757us) write_op.cc:312] APPLY: finished
0114 08:59:13.068511 (+ 4907us) log.cc:844] Serialized 58 byte log entry
0114 08:59:13.068689 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:13.068804 (+ 115us) write_op.cc:454] Released schema lock
0114 08:59:13.068850 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:13.069209 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:13.069240 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5695434,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":136582,"apply.run_cpu_time_us":4554,"apply.run_wall_time_us":211879,"num_ops":7,"prepare.queue_time_us":402321,"prepare.run_cpu_time_us":46425,"prepare.run_wall_time_us":240556,"raft.queue_time_us":101915,"raft.run_cpu_time_us":5720,"raft.run_wall_time_us":61075,"replication_time_us":476797,"thread_start_us":546175,"threads_started":4,"wal-append.queue_time_us":264005,"wal-append.run_cpu_time_us":24319,"wal-append.run_wall_time_us":1594213}]]}
I20250114 08:59:14.589768 21555 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 08:59:14.590108 21546 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:14.593173 23480 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:14.593539 21702 raft_consensus.cc:1270] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:14.594468 21698 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250114 08:59:14.594830 23400 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:14.595628 21704 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 08:59:14.595763 21699 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:14.596946 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:14.597332 23699 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20250114 08:59:14.598273 22266 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:14.540839 23597 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250114 08:59:14.558501 23713 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.596546 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 132) took 1559 ms. Trace:
I20250114 08:59:14.576445 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.600625 21554 rpcz_store.cc:276] 0114 08:59:13.338229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.505141 (+1166912us) service_pool.cc:225] Handling call
0114 08:59:14.563419 (+ 58278us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.573154 (+ 9735us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.573349 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54528}
I20250114 08:59:14.603005 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:14.610340 21546 rpcz_store.cc:276] 0114 08:59:13.035518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.452256 (+1416738us) service_pool.cc:225] Handling call
0114 08:59:14.557911 (+105655us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.590527 (+ 32616us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.594926 (+ 4399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:14.603917 22274 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250114 08:59:14.621138 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:14.621408 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1366) took 3115 ms. Trace:
I20250114 08:59:14.584673 23614 consensus_queue.cc:1225] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250114 08:59:14.607087 23724 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.617035 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1297) took 2067 ms. Trace:
I20250114 08:59:14.628033 21555 rpcz_store.cc:276] 0114 08:59:12.549325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.425285 (+1875960us) service_pool.cc:225] Handling call
0114 08:59:14.439260 (+ 13975us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:14.590221 (+150961us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.616685 (+ 26464us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540007d0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:14.616947 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":345367936}
W20250114 08:59:14.611701 21596 leader_election.cc:336] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.764s (SENT)
I20250114 08:59:14.599545 22885 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.623349 21550 rpcz_store.cc:276] 0114 08:59:11.505150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.247841 (+2742691us) service_pool.cc:225] Handling call
0114 08:59:14.248203 (+ 362us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:14.248484 (+ 281us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:14.473020 (+224536us) log.cc:844] Serialized 14 byte log entry
0114 08:59:14.618738 (+145718us) log.cc:844] Serialized 14 byte log entry
0114 08:59:14.620066 (+ 1328us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:14.620130 (+ 64us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:14.620187 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.620266 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:14.620385 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41621,"prepare.queue_time_us":243859,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":7428,"raft.queue_time_us":41919,"thread_start_us":295802,"threads_started":2}
I20250114 08:59:14.641871 21703 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:14.655509 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 114413us
I20250114 08:59:14.664364 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:14.667146 23298 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250114 08:59:14.667873 23735 log.cc:926] Time spent T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250114 08:59:14.648658 23392 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.670156 23644 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:14.671811 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1290) took 2414 ms. Trace:
I20250114 08:59:14.673044 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:14.673497 21703 raft_consensus.cc:3049] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.673884 21703 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:14.674767 21703 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.677287 21703 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:14.675887 23493 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250114 08:59:14.683828 21547 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:14.599996 23472 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.698757 23472 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:14.678926 23502 consensus_meta.cc:287] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.534s user 0.004s sys 0.008s
I20250114 08:59:14.700341 23431 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:14.702296 21695 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:14.678299 23635 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
W20250114 08:59:14.679497 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 121134us
I20250114 08:59:14.705807 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:14.709020 23472 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:14.710497 23472 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.677323 21668 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:14.731135 21628 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
W20250114 08:59:14.731509 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 54389.6us
I20250114 08:59:14.666417 23449 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.698872 21700 raft_consensus.cc:3049] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.732889 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:14.734228 23502 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:14.734324 21668 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:14.734941 23502 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:14.708513 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:14.712477 23431 consensus_queue.cc:1046] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:14.713178 21594 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:14.713201 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:14.714900 21703 raft_consensus.cc:1270] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250114 08:59:14.720113 21552 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:14.672343 21548 rpcz_store.cc:276] 0114 08:59:12.257012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.420992 (+2163980us) service_pool.cc:225] Handling call
0114 08:59:14.460866 (+ 39874us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:14.497556 (+ 36690us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:14.497595 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:14.505998 (+ 8403us) log.cc:844] Serialized 26 byte log entry
0114 08:59:14.540920 (+ 34922us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:14.540961 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:14.552383 (+ 11422us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54003a16d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:14.552424 (+ 41us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:14.671362 (+118938us) raft_consensus.cc:1680] finished
0114 08:59:14.671400 (+ 38us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:14.671727 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":326,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":29982,"spinlock_wait_cycles":83397888,"thread_start_us":31924,"threads_started":1,"wal-append.queue_time_us":117980}
I20250114 08:59:14.691524 23713 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:14.733111 21700 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:14.709780 23537 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:14.735942 23596 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250114 08:59:14.711515 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 109756us
I20250114 08:59:14.743280 21698 raft_consensus.cc:1270] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250114 08:59:14.688040 23493 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:14.748339 23433 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.743315 21703 pending_rounds.cc:77] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 3
I20250114 08:59:14.774811 21703 pending_rounds.cc:99] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250114 08:59:14.774685 23433 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.775352 21703 raft_consensus.cc:2883] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250114 08:59:14.776983 21596 leader_election.cc:336] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.854s (SENT)
I20250114 08:59:14.777508 21596 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:14.779805 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 104724us
I20250114 08:59:14.780804 21669 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:14.781818 23726 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:14.782274 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:14.782311 23726 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:14.750542 21702 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:14.747851 23480 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:14.793669 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 80602.7us
I20250114 08:59:14.745493 23436 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:14.783053 23726 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.748222 23596 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.743041 23502 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.749845 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:14.790998 23449 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.802610 23225 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:14.804536 23298 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:14.801772 23713 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:14.812296 23298 raft_consensus.cc:695] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.812798 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.814072 23449 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.815209 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:14.819706 23672 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:14.820732 23510 consensus_queue.cc:1046] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:14.821797 23610 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:14.822785 23610 consensus_queue.cc:1225] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250114 08:59:14.812378 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:14.823450 23610 raft_consensus.cc:3049] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.823988 23610 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.824496 23709 consensus_queue.cc:1046] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:14.810038 21700 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.825094 23225 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:14.825469 23709 raft_consensus.cc:3049] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.825680 21700 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.825891 23709 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:14.822029 23510 consensus_queue.cc:1035] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 08:59:14.827000 23709 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.813827 23298 consensus_queue.cc:237] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.798641 23436 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.804802 23596 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:14.836993 23433 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.837246 23610 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.854166 23610 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.852738 23436 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:14.838840 23644 raft_consensus.cc:3049] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:14.840628 22267 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
W20250114 08:59:14.841427 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 64513.5us
I20250114 08:59:14.825505 23225 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.841444 23472 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:14.842187 23683 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.816284 23449 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:14.844547 23734 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:14.849041 23431 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:14.830828 23709 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:14.864449 21547 raft_consensus.cc:1270] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:14.865998 21621 raft_consensus.cc:1270] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:59:14.871589 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:14.880847 21703 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:14.873545 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:14.869976 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:14.852790 23713 consensus_queue.cc:1046] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250114 08:59:14.885838 21701 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:14.865404 23644 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:14.888792 21596 leader_election.cc:336] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.877s (SENT)
I20250114 08:59:14.878283 21700 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20250114 08:59:14.879462 23718 consensus_meta.cc:287] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.570s user 0.007s sys 0.004s
I20250114 08:59:14.892968 23225 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.897598 21593 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:14.897598 23436 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:14.890765 21700 pending_rounds.cc:77] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 2
I20250114 08:59:14.901209 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:14.905504 21700 pending_rounds.cc:99] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:14.915064 21700 raft_consensus.cc:2883] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:14.908123 23449 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:14.908057 23472 consensus_queue.cc:1046] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250114 08:59:14.909098 23400 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:59:14.909915 23436 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:14.910770 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:14.924342 21628 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:14.904963 23431 consensus_queue.cc:1225] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:14.913239 21625 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:14.913605 21624 raft_consensus.cc:3049] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:14.925206 21624 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.913846 23610 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:14.914460 21622 raft_consensus.cc:3049] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
W20250114 08:59:14.907431 23746 log.cc:926] Time spent T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20250114 08:59:14.926566 21622 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.927141 21521 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:14.934753 21630 raft_consensus.cc:1270] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250114 08:59:14.934348 21624 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.917290 23672 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:14.935119 21630 pending_rounds.cc:77] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 2
I20250114 08:59:14.903465 22925 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20250114 08:59:14.896342 23502 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.905426 23597 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20250114 08:59:14.962471 21630 pending_rounds.cc:99] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:14.962944 21630 raft_consensus.cc:2883] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:14.904239 23718 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.983322 21624 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:14.988564 21625 raft_consensus.cc:3049] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:14.989095 21625 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:14.992391 21626 raft_consensus.cc:1270] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:14.948354 23614 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:14.955264 23672 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:14.959590 23480 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:14.923208 23449 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.978389 21622 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:14.935918 23635 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.047s
I20250114 08:59:14.990952 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:15.029309 21622 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:15.006937 23672 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:15.008139 23225 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:15.038103 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:15.053866 21630 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:15.001420 23614 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:15.018697 21625 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:15.055949 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 237493us
I20250114 08:59:15.035241 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:15.080780 21621 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:15.052969 23436 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:15.057209 23614 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:15.081889 21697 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:14.991204 23644 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.065461 23431 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.076757 22925 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
I20250114 08:59:15.037580 23392 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20250114 08:59:15.069989 23683 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.099723 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:15.101819 21521 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:15.104957 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:15.088035 23537 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:15.094765 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 290621us
I20250114 08:59:15.116798 21628 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:15.100997 23644 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:15.178031 21624 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:15.126157 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:15.159996 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:15.102984 23350 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.204322 21630 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:15.300882 21621 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:15.126909 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:15.145995 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 265569us
W20250114 08:59:15.195025 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 412788us
W20250114 08:59:15.302073 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 181212us
W20250114 08:59:15.153851 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 306028us
W20250114 08:59:15.293166 21596 leader_election.cc:336] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.577s (SENT)
I20250114 08:59:15.305604 21625 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:15.113283 22885 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250114 08:59:15.309558 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 188387us
I20250114 08:59:15.311265 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:15.314826 21596 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:15.310361 21521 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:15.282178 23751 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20250114 08:59:15.211223 23749 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250114 08:59:15.106055 23713 consensus_queue.cc:1225] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:15.143203 23702 consensus_queue.cc:1035] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.319s
I20250114 08:59:15.224609 23480 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.210932 23759 raft_consensus.cc:3049] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:15.103502 23436 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:15.340261 23759 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:15.341354 23759 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.192008 23449 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.154101 23718 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:15.342154 23436 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.164543 23726 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.124794 23690 consensus_queue.cc:1046] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.337s
I20250114 08:59:15.154232 23502 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:15.117945 23599 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:15.393090 23599 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:15.393581 23599 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:15.277989 22241 leader_election.cc:336] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.725s (SENT)
I20250114 08:59:15.312973 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:15.400295 21630 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:15.334728 21624 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:15.208223 23510 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.124827 23537 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.338569 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:15.419620 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:15.339100 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:15.339250 23756 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:15.313905 23350 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.251381 23635 consensus_queue.cc:1035] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.088s
I20250114 08:59:15.434080 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:15.339644 23435 consensus_queue.cc:1035] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20250114 08:59:15.151402 23400 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.165s
I20250114 08:59:15.316499 22885 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:15.325201 23754 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:15.106105 23431 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:15.381057 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:15.242098 23610 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.101s
W20250114 08:59:15.399608 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 138830us
I20250114 08:59:15.402662 21625 raft_consensus.cc:1270] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20250114 08:59:15.148937 23755 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.230s
I20250114 08:59:15.125331 23298 consensus_queue.cc:1046] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.257s
I20250114 08:59:15.420091 21629 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:15.432457 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 225282us
W20250114 08:59:15.387091 23596 consensus_meta.cc:287] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.534s user 0.012s sys 0.000s
W20250114 08:59:15.457137 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 76195us
I20250114 08:59:15.456230 21625 pending_rounds.cc:77] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 1
I20250114 08:59:15.461879 21625 pending_rounds.cc:99] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:15.462193 21625 raft_consensus.cc:2883] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:15.433400 21621 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:15.439287 23502 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:15.432008 22925 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.378540 23480 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.452347 22885 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:15.399849 23759 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:15.462565 21625 pending_rounds.cc:99] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:15.472853 21625 raft_consensus.cc:2883] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:15.440502 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:15.465060 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 129599us
W20250114 08:59:15.455334 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 148994us
I20250114 08:59:15.470523 22925 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.480243 23772 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:15.480778 23772 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:15.481117 23772 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:15.481757 22925 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.482481 21629 pending_rounds.cc:77] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 2
I20250114 08:59:15.482807 21629 pending_rounds.cc:99] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 08:59:15.483058 21629 raft_consensus.cc:2883] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:15.483646 21630 raft_consensus.cc:2405] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:15.490868 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:15.491644 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:15.462327 23754 raft_consensus.cc:695] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:15.429199 23510 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.498476 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:15.500715 23433 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
W20250114 08:59:15.501835 23745 log.cc:926] Time spent T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250114 08:59:15.502426 23431 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:15.503268 23758 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
I20250114 08:59:15.503701 21548 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:15.463683 23768 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:15.499569 21624 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:15.507685 21625 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:15.454486 23589 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20250114 08:59:15.508776 21547 raft_consensus.cc:2371] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7. Re-sending same reply.
I20250114 08:59:15.454058 23440 consensus_queue.cc:1035] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.234s
I20250114 08:59:15.505168 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:15.514591 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:15.505942 23709 consensus_meta.cc:287] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.614s user 0.012s sys 0.001s
I20250114 08:59:15.519332 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:15.440049 22241 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:15.519889 21669 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:15.521834 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:15.517078 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:15.464181 23596 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.497208 23449 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.526721 23431 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:15.525390 21704 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:15.524430 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:15.447849 23756 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:15.532181 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:15.529117 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 246214us
I20250114 08:59:15.517875 23734 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:15.535454 23672 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:15.547291 23672 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:15.543905 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:15.548287 21547 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:15.548815 21628 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250114 08:59:15.551578 23756 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:15.535459 21546 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:15.534353 21549 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:15.566597 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:15.567276 21668 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:15.570930 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 80135.3us
I20250114 08:59:15.532279 23768 raft_consensus.cc:695] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
W20250114 08:59:15.597537 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 142214us
I20250114 08:59:15.563938 23709 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:15.552690 23467 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20250114 08:59:15.567936 23449 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:15.696725 21594 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:15.796187 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 288184us
W20250114 08:59:15.663381 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 158208us
I20250114 08:59:15.799402 21547 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
W20250114 08:59:15.930917 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 392311us
I20250114 08:59:15.969756 21546 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:15.603693 21668 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:15.986435 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 323654us
W20250114 08:59:15.581368 23752 log.cc:926] Time spent T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.310s user 0.002s sys 0.000s
I20250114 08:59:15.996970 23786 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:15.997702 23786 raft_consensus.cc:695] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:15.788146 23775 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.078s user 0.002s sys 0.000s
I20250114 08:59:15.994653 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:15.597894 23756 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:15.869253 23781 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:15.576752 23726 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:15.869222 23782 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.270s user 0.000s sys 0.000s
I20250114 08:59:15.784984 23761 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 08:59:15.917666 23599 consensus_meta.cc:287] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.503s user 0.008s sys 0.004s
I20250114 08:59:16.000828 21704 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:15.995332 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:15.919059 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 336858us
I20250114 08:59:15.999423 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1602) took 1293 ms. Trace:
W20250114 08:59:15.786854 23778 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.259s user 0.000s sys 0.000s
I20250114 08:59:16.018321 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.053300 21699 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:15.993430 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 403734us
I20250114 08:59:15.865598 23768 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.023893 21702 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:15.786146 23771 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.279s user 0.000s sys 0.001s
W20250114 08:59:15.786731 23225 consensus_meta.cc:287] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.724s user 0.007s sys 0.004s
I20250114 08:59:15.589210 23614 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.931044 23472 raft_consensus.cc:3049] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:15.889672 23610 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:15.901309 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:15.681406 23350 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.865486 23596 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:15.845454 23440 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.901090 23480 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:15.978376 23510 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.998267 23776 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.869976 23702 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:15.845400 23537 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:15.851943 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:16.001044 23690 raft_consensus.cc:3049] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:16.000262 23762 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.057456 21703 raft_consensus.cc:2405] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20250114 08:59:16.057371 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:16.020224 23756 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:16.019150 23759 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:15.868662 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:15.890277 23755 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.179s
W20250114 08:59:15.872445 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 627435us
I20250114 08:59:16.076138 21669 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:15.889066 23783 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.168s
W20250114 08:59:16.079102 21596 leader_election.cc:336] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.921s (SENT)
W20250114 08:59:16.080807 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 93623.4us
I20250114 08:59:16.054840 21696 rpcz_store.cc:276] 0114 08:59:14.705792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.736142 (+ 30350us) service_pool.cc:225] Handling call
0114 08:59:14.736680 (+ 538us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:14.737294 (+ 614us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:14.737335 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:14.738118 (+ 783us) log.cc:844] Serialized 52 byte log entry
0114 08:59:15.151939 (+413821us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:15.151993 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:15.152084 (+ 91us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:15.314623 (+162539us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:15.998973 (+684350us) raft_consensus.cc:1680] finished
0114 08:59:15.998993 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:15.999242 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1298,"raft.queue_time_us":1484,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":41014,"spinlock_wait_cycles":26595712,"thread_start_us":413325,"threads_started":1,"wal-append.queue_time_us":472955}
I20250114 08:59:16.101519 23796 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.076555 23610 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:16.228910 23785 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.228804 23610 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.229468 23785 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.230195 23610 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:16.051366 23784 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.230540 23779 raft_consensus.cc:3049] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:16.230957 23779 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:16.230912 23784 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.231802 23785 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.109498 23801 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.216663 23510 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:16.233207 23780 log.cc:926] Time spent T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.448s user 0.001s sys 0.000s
I20250114 08:59:16.232970 23801 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.071080 23774 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.078140 23472 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:16.041960 23781 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:15.600179 23754 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.070885 23433 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.071s
I20250114 08:59:16.051452 23599 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:16.223199 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 165939us
W20250114 08:59:16.150620 23772 consensus_meta.cc:287] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.669s user 0.012s sys 0.004s
I20250114 08:59:16.227278 21627 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250114 08:59:16.233028 23510 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.106405 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:16.149016 23225 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.210757 23690 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:16.222934 23756 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:16.210553 22267 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
W20250114 08:59:16.228799 23795 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20250114 08:59:16.137455 23440 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.234493 23596 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:16.285388 23298 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:16.306208 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 227368us
W20250114 08:59:16.303658 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 91679.9us
I20250114 08:59:16.302811 21668 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:16.293242 23759 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:16.231958 23779 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.298426 23480 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 08:59:16.148736 23702 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:16.311062 21520 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.663s (SENT) [suppressed 45 similar messages]
I20250114 08:59:16.322834 21547 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:16.323923 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.326982 21553 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:16.328058 21548 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20250114 08:59:16.306748 23350 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.333087 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:16.333278 23815 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.342978 21551 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: [2.2-4.3]
W20250114 08:59:16.343158 21596 leader_election.cc:336] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.977s (SENT)
I20250114 08:59:16.344489 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:16.344784 21548 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:16.334391 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.252260 23796 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:16.355049 21596 leader_election.cc:336] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.053s (SENT)
I20250114 08:59:16.355142 23802 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.355604 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.355818 23802 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:16.356535 21596 leader_election.cc:336] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.138s (SENT)
W20250114 08:59:16.358084 23777 log.cc:926] Time spent T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.798s user 0.000s sys 0.010s
W20250114 08:59:16.359155 22885 consensus_meta.cc:287] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.859s user 0.009s sys 0.000s
I20250114 08:59:16.224344 23762 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.223532 23786 consensus_queue.cc:237] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.355132 21551 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:16.363488 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 327895us
I20250114 08:59:16.149207 23776 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:16.374226 22241 leader_election.cc:336] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.639s (SENT)
I20250114 08:59:16.377785 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:16.314555 23449 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.369671 23599 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.370926 23440 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.314999 23298 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.373461 23759 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.368516 23690 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.382398 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.396750 21547 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:16.380201 21594 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:16.359551 22885 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.363320 23480 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:16.296864 23772 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:16.391423 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 88706.3us
I20250114 08:59:16.300738 23225 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.285455 23774 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.363451 23614 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.406884 21594 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:16.399309 23480 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:16.411427 23431 consensus_meta.cc:287] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.850s user 0.013s sys 0.000s
I20250114 08:59:16.411619 23510 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.437909 21594 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:16.393977 23779 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:16.411727 23793 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.411868 23502 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.413895 23772 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.414381 22269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 183) took 1585 ms. Trace:
I20250114 08:59:16.416126 23225 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:16.416532 23767 log.cc:926] Time spent T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 1.105s user 0.004s sys 0.000s
I20250114 08:59:16.386157 23350 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.416635 23788 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250114 08:59:16.417419 23644 consensus_meta.cc:287] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.125s user 0.012s sys 0.000s
W20250114 08:59:16.417747 23805 log.cc:926] Time spent T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20250114 08:59:16.406658 23433 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250114 08:59:16.412218 23440 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20250114 08:59:16.418727 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:16.506412 22271 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:16.506999 23350 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.508597 23806 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:16.509109 23806 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:16.509502 23806 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:16.509886 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:16.510568 22266 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:16.511835 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:16.480530 23793 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.419548 23784 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.420718 21668 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:16.363869 23815 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.425348 21554 raft_consensus.cc:1235] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 08:59:16.425535 21555 raft_consensus.cc:2363] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7.
I20250114 08:59:16.422606 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1449) took 1193 ms. Trace:
I20250114 08:59:16.442692 23431 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:16.517823 21621 rpcz_store.cc:276] 0114 08:59:15.228708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.479851 (+251143us) service_pool.cc:225] Handling call
0114 08:59:15.480285 (+ 434us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:15.937361 (+457076us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:15.937405 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:15.937904 (+ 499us) log.cc:844] Serialized 26 byte log entry
0114 08:59:16.068228 (+130324us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:16.068267 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:16.068340 (+ 73us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.422307 (+353967us) raft_consensus.cc:1680] finished
0114 08:59:16.422341 (+ 34us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.422521 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76966,"raft.queue_time_us":122179,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":33749,"thread_start_us":524595,"threads_started":2,"wal-append.queue_time_us":338173}
I20250114 08:59:16.416217 23690 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:16.437390 23480 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:16.482147 23788 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:16.484074 23502 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.419262 23802 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:16.515864 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 108237us
I20250114 08:59:16.473938 23796 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:16.398030 21596 connection.cc:232] RPC call timeout handler was delayed by -1.1283043569999336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20250114 08:59:16.518677 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1425) took 1606 ms. Trace:
I20250114 08:59:16.364904 23472 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.480820 23510 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
W20250114 08:59:16.529145 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 72935us
W20250114 08:59:16.536425 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 147840us
I20250114 08:59:16.543267 23644 raft_consensus.cc:3049] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:16.544669 23803 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:16.546244 23762 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:16.547536 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1620) took 1058 ms. Trace:
I20250114 08:59:16.427233 23783 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.090s
W20250114 08:59:16.520599 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 99974us
W20250114 08:59:16.563786 21596 connection.cc:232] RPC call timeout handler was delayed by -1.0024249949999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21200000000000002s for a response
I20250114 08:59:16.425976 23596 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.550906 23824 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:16.573000 21669 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:16.564185 21623 rpcz_store.cc:276] 0114 08:59:14.912235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.914014 (+ 1779us) service_pool.cc:225] Handling call
0114 08:59:14.914385 (+ 371us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:14.935533 (+ 21148us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:14.935564 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:14.953759 (+ 18195us) log.cc:844] Serialized 26 byte log entry
0114 08:59:15.117631 (+163872us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:15.118170 (+ 539us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:15.118272 (+ 102us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.415850 (+1297578us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:16.513292 (+ 97442us) raft_consensus.cc:1680] finished
0114 08:59:16.513315 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.518584 (+ 5269us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16024,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":21728,"spinlock_wait_cycles":6111104,"thread_start_us":101345,"threads_started":1,"wal-append.queue_time_us":289163}
I20250114 08:59:16.515414 23801 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.571928 21521 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:16.583599 21668 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:16.575959 21700 rpcz_store.cc:276] 0114 08:59:15.489006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.489153 (+ 147us) service_pool.cc:225] Handling call
0114 08:59:15.489499 (+ 346us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:15.499251 (+ 9752us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:15.499290 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:15.525718 (+ 26428us) log.cc:844] Serialized 52 byte log entry
0114 08:59:15.988582 (+462864us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:15.988710 (+ 128us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:15.988862 (+ 152us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.547304 (+558442us) raft_consensus.cc:1680] finished
0114 08:59:16.547326 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.547472 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6250,"raft.queue_time_us":16646,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":3797,"thread_start_us":413011,"threads_started":1,"wal-append.queue_time_us":672224}
I20250114 08:59:16.552279 23788 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:16.588898 23796 raft_consensus.cc:3049] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:16.589587 23796 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:16.590128 23502 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.472244 22269 rpcz_store.cc:276] 0114 08:59:14.829239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.840514 (+ 11275us) service_pool.cc:225] Handling call
0114 08:59:14.869784 (+ 29270us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:15.298957 (+429173us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:15.308656 (+ 9699us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:15.337974 (+ 29318us) log.cc:844] Serialized 26 byte log entry
0114 08:59:15.490213 (+152239us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:15.495714 (+ 5501us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:15.495811 (+ 97us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.094715 (+598904us) raft_consensus.cc:1680] finished
0114 08:59:16.094744 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.414070 (+319326us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5400720450. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:16.414293 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18791,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":54108544,"thread_start_us":335613,"threads_started":2,"wal-append.queue_time_us":172067}
I20250114 08:59:16.591581 23781 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.590999 23796 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.591820 23449 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:16.592108 23596 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.593065 23772 raft_consensus.cc:3049] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:16.593472 23772 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:16.594300 23772 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.594321 23596 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:16.591388 23812 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:16.596407 23812 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:16.596866 23812 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:16.597334 22241 leader_election.cc:336] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.085s (SENT)
I20250114 08:59:16.597987 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:16.599584 22241 leader_election.cc:336] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.234s (SENT)
I20250114 08:59:16.572921 23802 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.600972 23802 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.603514 23802 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.604264 23762 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:16.604993 23762 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:16.605660 23768 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:16.606227 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:16.605952 23762 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.606904 22269 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 2.
I20250114 08:59:16.611346 23772 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:16.613395 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1445) took 1386 ms. Trace:
I20250114 08:59:16.613937 21629 rpcz_store.cc:276] 0114 08:59:15.227339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.459299 (+231960us) service_pool.cc:225] Handling call
0114 08:59:15.471115 (+ 11816us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:15.915355 (+444240us) spinlock_profiling.cc:243] Waited 6.65 ms on lock 0x7b48000ce358. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa644491581 00007fa644488677 00007fa64448850a 00007fa64457c9bf 00007fa64457c687 00007fa644590b8c 00007fa644590b38 00007fa644590ab8 00007fa644590a73 00007fa64458fd1c 00007fa6434fcd16
0114 08:59:15.959228 (+ 43873us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:15.959277 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:15.991342 (+ 32065us) log.cc:844] Serialized 52 byte log entry
0114 08:59:16.301180 (+309838us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:16.301230 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:16.301333 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.612952 (+311619us) raft_consensus.cc:1680] finished
0114 08:59:16.612983 (+ 31us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.613310 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13381,"raft.queue_time_us":288855,"raft.run_cpu_time_us":2177,"raft.run_wall_time_us":70088,"spinlock_wait_cycles":15300864,"thread_start_us":678627,"threads_started":2,"wal-append.queue_time_us":395468}
I20250114 08:59:16.615656 23774 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.619335 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 137) took 1125 ms. Trace:
I20250114 08:59:16.622270 23702 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:16.622354 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 54739.3us
W20250114 08:59:16.623015 23799 log.cc:926] Time spent T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.479s user 0.002s sys 0.000s
I20250114 08:59:16.623656 21669 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:16.624634 22885 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.627605 23800 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:16.628197 23776 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.629406 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:16.630632 23800 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:16.631251 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1432) took 1690 ms. Trace:
I20250114 08:59:16.631668 21627 rpcz_store.cc:276] 0114 08:59:14.940654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:14.940807 (+ 153us) service_pool.cc:225] Handling call
0114 08:59:14.942159 (+ 1352us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:16.286990 (+1344831us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:16.631006 (+344016us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400165dd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:16.631186 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2624002304}
I20250114 08:59:16.571779 23644 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:16.633062 23726 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.633262 23644 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.633827 23726 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.634649 23644 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:16.637261 23298 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.637457 23800 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.637962 23823 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.638017 23800 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.567495 23431 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.600127 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:16.589076 21596 leader_election.cc:336] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.584s (SENT)
I20250114 08:59:16.604959 21521 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:16.637974 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1448) took 1409 ms. Trace:
W20250114 08:59:16.675084 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 237279us
I20250114 08:59:16.646153 21628 rpcz_store.cc:276] 0114 08:59:15.228361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.470252 (+241891us) service_pool.cc:225] Handling call
0114 08:59:15.495358 (+ 25106us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:15.549243 (+ 53885us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:16.637694 (+1088451us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b54001518d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:16.637901 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122277632}
I20250114 08:59:16.686864 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:16.688073 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 83206.8us
I20250114 08:59:16.688337 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 140) took 1184 ms. Trace:
I20250114 08:59:16.688730 21550 rpcz_store.cc:276] 0114 08:59:15.504410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.513087 (+ 8677us) service_pool.cc:225] Handling call
0114 08:59:15.521785 (+ 8698us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:15.553005 (+ 31220us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:15.553044 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:15.563347 (+ 10303us) log.cc:844] Serialized 26 byte log entry
0114 08:59:15.912906 (+349559us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:16.073827 (+160921us) log.cc:844] Serialized 14 byte log entry
0114 08:59:16.215946 (+142119us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:16.216029 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.622409 (+406380us) spinlock_profiling.cc:243] Waited 61.5 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:16.641285 (+ 18876us) raft_consensus.cc:1680] finished
0114 08:59:16.688111 (+ 46826us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.688280 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11042,"prepare.queue_time_us":461494,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":4620,"raft.queue_time_us":199296,"raft.run_wall_time_us":67871,"spinlock_wait_cycles":185057152,"thread_start_us":488721,"threads_started":3,"wal-append.queue_time_us":487966}
I20250114 08:59:16.689529 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.687798 21551 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:16.691850 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.639369 23431 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.619752 21552 rpcz_store.cc:276] 0114 08:59:15.494224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.505706 (+ 11482us) service_pool.cc:225] Handling call
0114 08:59:15.517488 (+ 11782us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:16.299797 (+782309us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:16.299952 (+ 155us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:16.319946 (+ 19994us) log.cc:844] Serialized 26 byte log entry
0114 08:59:16.524046 (+204100us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:16.524095 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:16.618917 (+ 94822us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b540008e154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:16.618963 (+ 46us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:16.619009 (+ 46us) raft_consensus.cc:1680] finished
0114 08:59:16.619022 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:16.619254 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21795,"raft.queue_time_us":179378,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":6143,"spinlock_wait_cycles":1180267904,"thread_start_us":388480,"threads_started":2,"wal-append.queue_time_us":204083}
I20250114 08:59:16.700510 23755 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:16.701333 23755 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.703943 23755 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:16.705297 23755 consensus_queue.cc:1046] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250114 08:59:16.722576 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 321496us
I20250114 08:59:16.725230 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:16.727559 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.729998 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.730063 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.706446 23783 raft_consensus.cc:3049] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Stepping down as leader of term 2
I20250114 08:59:16.694577 23796 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:16.687341 23726 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:16.694953 23824 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:16.722640 23472 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:16.739069 21520 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:16.725538 23596 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 08:59:16.744047 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 114816us
I20250114 08:59:16.731845 23754 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:16.638684 23823 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:16.745838 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 156786us
I20250114 08:59:16.747689 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:16.745213 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 145687us
I20250114 08:59:16.749797 21594 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:16.750582 23787 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:16.750855 23800 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:16.743070 23610 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.510s user 0.008s sys 0.004s
I20250114 08:59:16.756682 23724 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:16.757212 23431 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.738288 23699 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:16.733973 23783 raft_consensus.cc:738] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:16.755300 23480 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.773036 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:16.780218 21695 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:16.723433 23812 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.766283 23787 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:16.843556 21596 leader_election.cc:336] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.699s (SENT)
I20250114 08:59:16.765338 23759 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:16.840783 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.773741 23786 raft_consensus.cc:2798] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 08:59:16.853408 21596 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:16.844552 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:16.851481 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 78209.5us
I20250114 08:59:16.854151 21695 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:16.858223 21698 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:16.858175 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.858860 21550 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:16.859948 21701 raft_consensus.cc:2363] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:16.862748 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:16.863201 21699 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:16.855696 23786 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:16.850112 23787 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:16.864446 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:16.868640 21702 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 6
I20250114 08:59:16.871304 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.871798 21703 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:16.875448 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.875994 21698 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:16.839437 23537 consensus_queue.cc:1035] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 08:59:16.864159 23786 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:16.854760 21551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:16.921283 21551 raft_consensus.cc:604] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:16.775521 23806 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:16.919649 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:16.924177 21703 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:16.925355 23806 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:16.931267 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:16.932217 21695 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:16.931514 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.937086 21704 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:16.938401 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.939849 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.941450 21704 raft_consensus.cc:2405] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 08:59:16.953830 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 175499us
I20250114 08:59:16.870698 23779 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:16.855396 23759 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:16.960769 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.962774 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.963418 21698 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 2.
I20250114 08:59:16.870291 23815 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:16.998430 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 213552us
I20250114 08:59:16.999435 21695 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:17.009553 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:17.022341 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.046324 21696 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.049728 21627 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:17.051297 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:16.859980 23824 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.011380 23350 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:17.066440 21703 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:17.041368 23844 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:16.869863 23783 consensus_queue.cc:260] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.060132 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.095139 21696 raft_consensus.cc:2405] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250114 08:59:17.083673 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 215328us
I20250114 08:59:17.057065 23818 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:17.097319 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.062933 23762 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:17.067363 23834 log.cc:926] Time spent T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
I20250114 08:59:17.059178 23811 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:17.113539 21703 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.089952 23812 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:17.065567 23788 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.113396 21698 raft_consensus.cc:1212] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[5.2-5.2] Dedup: 5.2->[]
I20250114 08:59:17.144639 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:17.146687 21698 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20250114 08:59:17.146728 21553 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:17.118306 23823 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:17.069536 23350 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.115192 23818 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:17.162209 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.163753 21695 raft_consensus.cc:2405] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 6 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:17.158332 23480 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:17.207122 21553 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:17.158990 23596 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:17.169055 23818 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:17.165584 21702 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:17.087927 23844 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:17.150579 23783 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 2 FOLLOWER]: Advancing to term 3
W20250114 08:59:17.170152 23690 consensus_meta.cc:287] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.606s user 0.007s sys 0.003s
I20250114 08:59:17.149096 23811 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:17.222584 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:17.216189 23772 consensus_meta.cc:287] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.604s user 0.012s sys 0.000s
I20250114 08:59:17.121299 23762 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:17.242368 21699 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:17.324990 21698 raft_consensus.cc:2405] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:17.290256 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:17.153462 23756 consensus_meta.cc:287] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.836s user 0.008s sys 0.003s
W20250114 08:59:17.215206 23795 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250114 08:59:17.336239 21702 raft_consensus.cc:2363] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
W20250114 08:59:17.166972 23644 consensus_meta.cc:287] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.532s user 0.011s sys 0.000s
W20250114 08:59:17.292395 23848 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20250114 08:59:17.341881 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.352628 21695 raft_consensus.cc:2388] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.218648 23759 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:17.233336 23837 log.cc:926] Time spent T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.268s user 0.000s sys 0.001s
W20250114 08:59:17.345350 23849 log.cc:926] Time spent T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20250114 08:59:17.345795 23844 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:17.348101 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.349812 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.289345 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:17.301465 23839 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.337s user 0.003s sys 0.000s
W20250114 08:59:17.325040 23794 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.004s sys 0.013s
I20250114 08:59:17.354667 21629 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250114 08:59:17.213583 23596 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.365041 21695 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:17.366223 23596 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:17.292598 23796 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:17.367329 23796 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.367997 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.369359 23796 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:17.370229 23856 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250114 08:59:17.351346 21699 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
I20250114 08:59:17.372128 21699 raft_consensus.cc:604] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:17.372498 21699 raft_consensus.cc:612] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:17.372754 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:17.373522 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.374115 21699 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:17.374158 23857 log.cc:926] Time spent T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20250114 08:59:17.233099 23788 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:17.363158 21702 raft_consensus.cc:1235] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3. Ops: []
I20250114 08:59:17.237447 23350 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:17.348390 23793 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:17.361018 23841 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.394s user 0.001s sys 0.000s
W20250114 08:59:17.378006 23794 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.004s sys 0.014s
W20250114 08:59:17.377451 23847 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.257s user 0.001s sys 0.000s
I20250114 08:59:17.396399 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.429828 21702 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:17.397286 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.398813 23762 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:17.401654 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:17.472713 22266 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:17.390776 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.342509 23756 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.410864 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:17.412009 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:17.418495 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46386 (request call id 209) took 1009 ms. Trace:
W20250114 08:59:17.425691 23472 consensus_meta.cc:287] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.679s user 0.007s sys 0.004s
I20250114 08:59:17.426362 23779 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:17.478247 21697 rpcz_store.cc:276] 0114 08:59:16.419342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:16.440440 (+ 21098us) service_pool.cc:225] Handling call
0114 08:59:17.417905 (+977465us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54002ea254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f552d50 00007fa63f5d5f2a 00007fa64452fc5e 00007fa63dc0c817 00007fa63dc0c7a2 00007fa63dc0c6d1 00007fa63dc0c64c 00007fa63dc0b902 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:17.418382 (+ 477us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":847967616}
I20250114 08:59:17.283007 23862 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.160s
I20250114 08:59:17.445884 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.489161 21701 raft_consensus.cc:2363] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:17.491299 21698 raft_consensus.cc:2363] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9.
I20250114 08:59:17.446079 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.447134 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:17.459787 23818 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:17.461766 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57308 (request call id 470) took 1103 ms. Trace:
I20250114 08:59:17.507503 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.382561 23644 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:17.465222 21699 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
W20250114 08:59:17.466707 23787 consensus_meta.cc:287] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.593s user 0.007s sys 0.001s
I20250114 08:59:17.467816 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.510226 21695 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:17.467595 23786 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:17.476006 23793 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:17.475989 21696 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:17.523852 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.524166 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:17.484973 23837 log.cc:926] Time spent T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20250114 08:59:17.521718 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.524765 21696 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:17.509446 21626 rpcz_store.cc:276] 0114 08:59:16.358114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:16.372547 (+ 14433us) service_pool.cc:225] Handling call
0114 08:59:17.461385 (+1088838us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54003b9cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d37c5 00007fa63f5cbc1a 00007fa64452c6c9 00007fa63dbfc194 00007fa63dbfc122 00007fa63dbfc051 00007fa63dbfbfcc 00007fa63dbfb282 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:17.461652 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112674688}
I20250114 08:59:17.526263 21697 raft_consensus.cc:2363] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:17.527423 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.528295 21704 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:17.530422 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:17.531322 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.531771 21628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:55318
I20250114 08:59:17.531960 21696 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:17.532444 21628 raft_consensus.cc:604] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:17.532871 21628 raft_consensus.cc:612] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:17.533028 23866 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:17.533784 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.533540 23866 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.534852 21702 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.535720 23866 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:17.535910 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.536505 21699 raft_consensus.cc:2363] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:17.485471 23811 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:17.542351 21702 raft_consensus.cc:2405] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:17.542604 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.548285 21695 raft_consensus.cc:2371] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5. Re-sending same reply.
I20250114 08:59:17.544983 21696 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:17.488417 22267 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:17.558573 21703 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
W20250114 08:59:17.558312 21596 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.010s sys 0.005s
W20250114 08:59:17.568408 21596 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.554s user 0.010s sys 0.005s
W20250114 08:59:17.573091 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 559342us
I20250114 08:59:17.611217 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.612083 21698 raft_consensus.cc:2363] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 2. Current term is 3.
I20250114 08:59:17.615181 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.616273 21696 raft_consensus.cc:2388] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.616081 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:17.538054 23830 log.cc:926] Time spent T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250114 08:59:17.616750 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 52729.1us
I20250114 08:59:17.622645 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.624981 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:17.628753 21593 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:17.520777 23644 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.519699 23787 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:17.547626 23793 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:17.643790 21596 leader_election.cc:336] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.758s (SENT)
I20250114 08:59:17.845276 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.672865 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.847293 21696 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.673698 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:17.805859 22241 leader_election.cc:336] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.503s (SENT)
W20250114 08:59:17.767982 21555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 139) took 2273 ms (client timeout 2227 ms). Trace:
W20250114 08:59:17.848819 21555 rpcz_store.cc:269] 0114 08:59:15.494844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:15.494998 (+ 154us) service_pool.cc:225] Handling call
0114 08:59:17.767716 (+2272718us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54001171d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:17.767908 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2066848512}
I20250114 08:59:17.642735 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.850257 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:17.850509 21695 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:17.767647 21626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "57f9cc1dd3c64647848a33a65b08c893"
from {username='slave'} at 127.0.0.1:55318
W20250114 08:59:17.666015 21521 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.007s sys 0.021s
I20250114 08:59:17.851898 21626 raft_consensus.cc:604] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Received request to transfer leadership to TS 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:17.852521 21521 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.806s user 0.013s sys 0.026s
I20250114 08:59:17.852878 21626 raft_consensus.cc:612] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250114 08:59:17.853353 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 843838us
I20250114 08:59:17.847215 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:17.633141 21704 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:17.861269 21553 raft_consensus.cc:2388] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.857079 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:17.869072 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.870573 21704 raft_consensus.cc:2405] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:17.864305 21700 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:17.872030 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.873584 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.876111 21702 raft_consensus.cc:2405] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 6 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20250114 08:59:17.740540 23837 log.cc:926] Time spent T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20250114 08:59:17.691747 23794 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:17.691993 23672 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:17.882941 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:17.629998 22266 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
W20250114 08:59:17.884141 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 78355.7us
I20250114 08:59:17.885007 21698 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:17.882870 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:17.740794 23431 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:17.916599 21695 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 08:59:17.885571 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.021605 21696 raft_consensus.cc:2371] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5. Re-sending same reply.
I20250114 08:59:18.022156 21702 raft_consensus.cc:2371] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6. Re-sending same reply.
W20250114 08:59:17.872256 23828 log.cc:926] Time spent T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250114 08:59:18.030037 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.030975 21703 raft_consensus.cc:2371] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5. Re-sending same reply.
W20250114 08:59:17.966187 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 111871us
I20250114 08:59:18.032145 21521 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.023919 21700 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:17.884176 21704 raft_consensus.cc:1212] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
W20250114 08:59:18.023908 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 380149us
I20250114 08:59:17.884142 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250114 08:59:18.045215 21695 raft_consensus.cc:1270] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:18.057092 21704 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250114 08:59:18.051769 21699 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:18.061844 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:18.062623 21547 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:18.068024 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:18.028861 23798 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.800s user 0.006s sys 0.001s
I20250114 08:59:18.074586 21696 raft_consensus.cc:2405] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:18.075160 21521 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.077021 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.077901 21702 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:18.070725 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:18.035748 23768 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.029s sys 0.028s
W20250114 08:59:18.072386 21596 leader_election.cc:336] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.920s (SENT)
I20250114 08:59:18.081247 21596 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:17.908778 23806 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:18.079996 21703 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:17.942605 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:17.909304 23472 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:18.082751 21623 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.020s sys 0.065s
I20250114 08:59:18.022228 23794 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:17.886358 23756 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:17.984017 23818 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:18.088795 21623 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.021s sys 0.066s
I20250114 08:59:17.952908 23597 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:18.063910 23537 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.068746 22885 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:18.096503 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:18.084159 23768 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.031s sys 0.030s
I20250114 08:59:18.101006 21698 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:18.081346 23690 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.109982 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.100214 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.113157 21703 raft_consensus.cc:2363] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9.
I20250114 08:59:18.089304 22269 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 7
W20250114 08:59:18.118361 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 46013.5us
I20250114 08:59:18.092908 23794 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:18.094699 23755 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:18.060866 23672 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.059267 23431 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.103938 22885 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:18.102605 23822 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.145583 23822 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.135193 23672 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.153029 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.136947 23431 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.139200 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.140141 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.140637 22885 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:18.129189 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:18.128180 23793 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.154843 21553 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:18.094834 23472 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.099726 23786 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:18.113129 23537 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.116559 23690 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.148365 23822 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.161824 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1405) took 1813 ms. Trace:
I20250114 08:59:18.108054 23597 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:18.173660 21548 rpcz_store.cc:276] 0114 08:59:16.348860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:16.351431 (+ 2571us) service_pool.cc:225] Handling call
0114 08:59:16.367898 (+ 16467us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:16.395604 (+ 27706us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:16.395633 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:16.447848 (+ 52215us) log.cc:844] Serialized 52 byte log entry
0114 08:59:16.674997 (+227149us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:16.675034 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:18.159636 (+1484602us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5400194e54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:18.159688 (+ 52us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:18.159744 (+ 56us) raft_consensus.cc:1680] finished
0114 08:59:18.159769 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:18.161750 (+ 1981us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9470,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":7008,"spinlock_wait_cycles":26675072,"thread_start_us":129871,"threads_started":1,"wal-append.queue_time_us":264050}
I20250114 08:59:18.132818 23702 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:18.165681 22266 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:18.172266 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.160533 21547 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:18.179997 21628 raft_consensus.cc:2405] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:18.159442 23644 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.164297 23787 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.190132 23537 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:18.193694 21596 leader_election.cc:336] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.955s (SENT)
I20250114 08:59:18.194360 21596 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:18.179353 23702 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:18.198539 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:18.202409 21628 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
W20250114 08:59:18.217409 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1246) took 1299 ms (client timeout 1577 ms). Trace:
I20250114 08:59:18.200582 23702 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:18.203343 23644 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 08:59:18.218006 21701 rpcz_store.cc:269] 0114 08:59:16.918405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:16.953256 (+ 34851us) service_pool.cc:225] Handling call
0114 08:59:18.217150 (+1263894us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b5400328f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:18.217342 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1213039488}
W20250114 08:59:18.229562 23879 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20250114 08:59:18.209189 23786 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250114 08:59:18.248093 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 64479.5us
I20250114 08:59:18.256315 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.242250 23786 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:18.254956 23852 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.258806 23755 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.278227 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.280508 21626 raft_consensus.cc:2405] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250114 08:59:18.276891 21596 leader_election.cc:336] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.251s (SENT)
I20250114 08:59:18.281463 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:18.282773 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.265223 23786 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:18.287523 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.296856 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.297816 21629 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:18.298820 22269 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:18.299775 23597 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:18.299823 22269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49020 (request call id 190) took 1654 ms (client timeout 1628 ms). Trace:
I20250114 08:59:18.276135 23472 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:18.275283 23690 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:18.320501 22241 leader_election.cc:336] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.551s (SENT)
I20250114 08:59:18.282135 23852 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:18.319561 23610 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 08:59:18.337965 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 74812.9us
I20250114 08:59:18.285624 23755 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.333822 23472 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.382930 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.383836 21621 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:18.385180 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.388689 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:18.300872 22269 rpcz_store.cc:269] 0114 08:59:16.645547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:16.684681 (+ 39134us) service_pool.cc:225] Handling call
0114 08:59:17.942478 (+1257797us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656d0e 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:18.299691 (+357213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12547456}
I20250114 08:59:18.395669 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.396334 21623 raft_consensus.cc:2405] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250114 08:59:18.396893 21628 raft_consensus.cc:2405] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250114 08:59:18.402312 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.403818 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:18.404645 21548 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.404932 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.405714 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:18.395342 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 320274us
I20250114 08:59:18.365013 23822 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.411702 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.413162 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.413822 21629 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:18.415269 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.378244 23784 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.376132 23796 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:18.416024 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 148285us
I20250114 08:59:18.371104 23756 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.417412 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.421249 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.422540 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.424407 21551 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20250114 08:59:18.426371 21628 raft_consensus.cc:2405] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:18.427251 23852 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:18.427400 23755 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.428242 23756 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.430192 23756 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:18.431273 23794 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:18.432926 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:18.433214 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.433918 21552 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:18.435014 21554 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:18.436051 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:18.418265 23822 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.436682 21546 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.436863 21555 raft_consensus.cc:2388] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 3: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.433260 21549 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
W20250114 08:59:18.438477 21520 connection.cc:232] RPC call timeout handler was delayed by -1.1828365459999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
I20250114 08:59:18.439316 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.442821 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:18.448220 21553 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.449213 21629 raft_consensus.cc:2363] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:18.449273 21521 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.461145 21548 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:18.468456 21553 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:18.469347 21630 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:18.470129 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.470999 21547 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:18.413308 23472 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.480211 21521 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.476886 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.478899 21546 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:18.482102 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.482770 21551 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:18.483880 21668 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.451428 23868 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.489308 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:18.489881 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.492269 21521 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.496621 21629 raft_consensus.cc:2388] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.424417 23796 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.422787 23784 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.472230 23449 raft_consensus.cc:2758] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250114 08:59:18.504850 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.490767 23597 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.495265 23589 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:18.515697 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.499053 23824 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:18.503866 23801 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:18.483904 23431 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20250114 08:59:18.519801 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:18.513752 23755 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:18.523360 23756 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250114 08:59:18.520506 23874 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.525730 23824 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:18.500140 23868 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.533124 23589 raft_consensus.cc:695] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:18.566076 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.569999 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.580590 21628 raft_consensus.cc:2405] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:18.535440 23801 raft_consensus.cc:695] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:18.524657 23597 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.538627 23794 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:18.541090 23755 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:18.584666 21621 raft_consensus.cc:2405] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250114 08:59:18.593294 21520 leader_election.cc:336] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.587s (SENT)
I20250114 08:59:18.577064 23824 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.623975 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.626744 21628 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.624940 23755 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:18.636111 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:18.625073 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 127652us
W20250114 08:59:18.636036 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 138403us
I20250114 08:59:18.624747 23832 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:18.625562 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:18.648403 21628 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:18.641535 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:18.648869 21520 leader_election.cc:336] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.663s (SENT)
I20250114 08:59:18.655228 21520 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:18.626737 23822 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.655718 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.657801 21628 raft_consensus.cc:2405] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:18.659974 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.577827 23874 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:18.653378 23832 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:18.667212 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.673275 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.674707 23597 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.657449 23589 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:18.676038 23784 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.754205 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.755079 21621 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:18.761478 23755 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.684579 23801 consensus_queue.cc:237] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.764590 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.756078 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:18.690934 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 49512.3us
W20250114 08:59:18.691635 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 162633us
W20250114 08:59:18.683224 22885 consensus_meta.cc:287] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.518s user 0.010s sys 0.000s
I20250114 08:59:18.699445 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.768005 21623 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.703898 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:18.707819 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b369b35736c413ca069db061ed36bc5 with delete_type TABLET_DATA_TOMBSTONED (TS 28a83b7a7d9d4aa899b425086aa45a8e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33868
W20250114 08:59:18.708550 21520 leader_election.cc:336] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.739s (SENT)
W20250114 08:59:18.703450 21451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 73) took 13116 ms (client timeout 15000 ms). Trace:
I20250114 08:59:18.769001 21520 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:18.718299 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.770402 21630 raft_consensus.cc:2363] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:59:18.730625 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.731177 23431 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:18.733162 23893 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.737075 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.658787 22271 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:18.739952 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:18.776670 21547 raft_consensus.cc:2388] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:18.657811 23472 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:18.762020 22266 consensus_meta.cc:287] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.576s user 0.007s sys 0.004s
I20250114 08:59:18.685973 23796 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:18.762779 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 112748us
I20250114 08:59:18.691203 23885 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:18.763664 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:18.715535 23886 log.cc:926] Time spent T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.265s user 0.000s sys 0.001s
I20250114 08:59:18.783429 23893 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.801067 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:18.802819 23893 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:18.803373 23893 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:18.803810 23893 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.766724 21521 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.766323 21621 raft_consensus.cc:2371] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7. Re-sending same reply.
I20250114 08:59:18.781783 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.786509 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.789728 21626 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:18.789122 21668 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.790405 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.789616 23472 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 08:59:18.818097 21622 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.818151 21628 raft_consensus.cc:2388] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.818557 21630 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
W20250114 08:59:18.804206 23702 consensus_meta.cc:287] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.563s user 0.008s sys 0.003s
I20250114 08:59:18.822628 21668 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.823611 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.825582 21668 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:18.832319 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.833253 21626 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:18.821635 23884 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
I20250114 08:59:18.836851 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:18.769210 21451 rpcz_store.cc:269] 0114 08:59:05.587685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:05.593277 (+ 5592us) service_pool.cc:225] Handling call
0114 08:59:18.703355 (+13110078us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:14.894237 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:14.894913 (+ 676us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:14.894969 (+ 56us) write_op.cc:435] Acquired schema lock
0114 08:59:14.894987 (+ 18us) tablet.cc:659] Decoding operations
0114 08:59:14.895894 (+ 907us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:14.896001 (+ 107us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:14.896029 (+ 28us) tablet.cc:682] Acquiring locks for 6 operations
0114 08:59:14.982617 (+ 86588us) tablet.cc:698] Row locks acquired
0114 08:59:14.991780 (+ 9163us) write_op.cc:260] PREPARE: finished
0114 08:59:15.026611 (+ 34831us) write_op.cc:270] Start()
0114 08:59:15.053948 (+ 27337us) write_op.cc:276] Timestamp: P: 1736845155008359 usec, L: 0
0114 08:59:15.064230 (+ 10282us) op_driver.cc:348] REPLICATION: starting
0114 08:59:15.154556 (+ 90326us) log.cc:844] Serialized 2597 byte log entry
0114 08:59:16.918126 (+1763570us) op_driver.cc:464] REPLICATION: finished
0114 08:59:17.349476 (+431350us) write_op.cc:301] APPLY: starting
0114 08:59:17.349570 (+ 94us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:17.349822 (+ 252us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:17.375499 (+ 25677us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:17.415052 (+ 39553us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:17.415412 (+ 360us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0114 08:59:17.415481 (+ 69us) write_op.cc:312] APPLY: finished
0114 08:59:17.415958 (+ 477us) log.cc:844] Serialized 52 byte log entry
0114 08:59:17.416162 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:17.416271 (+ 109us) write_op.cc:454] Released schema lock
0114 08:59:17.416333 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:17.416774 (+ 441us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:17.416802 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":924642,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":387216,"apply.run_cpu_time_us":4318,"apply.run_wall_time_us":67292,"num_ops":6,"prepare.queue_time_us":285383,"prepare.run_cpu_time_us":21755,"prepare.run_wall_time_us":539556,"raft.queue_time_us":476713,"raft.run_cpu_time_us":10654,"raft.run_wall_time_us":219659,"replication_time_us":1802378,"thread_start_us":889211,"threads_started":4,"wal-append.queue_time_us":299431,"wal-append.run_cpu_time_us":42471,"wal-append.run_wall_time_us":2952619}]]}
I20250114 08:59:18.845894 23874 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:18.847126 23902 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:18.828059 23702 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.840366 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.781876 22266 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:18.858068 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.869612 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.780110 22885 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.866767 21629 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:18.876699 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.782433 23431 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.879951 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.849045 23868 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:18.878140 21593 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:18.881366 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.884749 21629 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.887287 21622 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:18.888561 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.890933 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.889362 21622 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:18.821902 23894 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.097s
W20250114 08:59:18.893565 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 68061us
I20250114 08:59:18.893787 21555 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:18.893837 21629 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:18.895815 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.896265 21621 raft_consensus.cc:2371] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5. Re-sending same reply.
I20250114 08:59:18.896672 21629 raft_consensus.cc:2388] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.897925 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.899542 21628 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:18.898511 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.899823 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:18.889876 23786 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.583s user 0.011s sys 0.001s
I20250114 08:59:18.905280 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.906087 21630 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:18.908075 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.908712 21622 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:18.912643 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:18.917060 21546 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:18.910411 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.918440 21629 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.924876 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.926216 21623 raft_consensus.cc:2405] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 08:59:18.889938 23793 raft_consensus.cc:2798] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election won for term 3
I20250114 08:59:18.930521 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.934891 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.935587 21703 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 2.
I20250114 08:59:18.936148 21701 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250114 08:59:18.936815 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.937300 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.937885 21696 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:18.939330 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.939852 21696 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:18.940871 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.945214 21702 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:18.946550 21696 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:18.948037 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.948894 21701 raft_consensus.cc:2371] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4. Re-sending same reply.
I20250114 08:59:18.951473 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.952075 21698 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:18.952345 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.952844 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.953403 21703 raft_consensus.cc:2371] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7. Re-sending same reply.
I20250114 08:59:18.953794 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.952100 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.954756 21704 raft_consensus.cc:2363] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7.
I20250114 08:59:18.955303 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.955945 21702 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:18.895714 23868 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:18.962278 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.963021 21630 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:18.962265 21698 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:18.968277 21629 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:18.972958 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.973863 21628 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:18.975988 22241 leader_election.cc:336] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.783s (SENT)
I20250114 08:59:18.980227 21546 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:18.981689 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:18.982533 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:18.982702 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:18.983161 21698 raft_consensus.cc:2371] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7. Re-sending same reply.
I20250114 08:59:18.984050 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.984220 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:18.984860 21630 raft_consensus.cc:2363] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
I20250114 08:59:18.985509 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:18.989095 23898 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250114 08:59:18.989435 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:18.990578 21623 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:18.928320 23786 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.993433 22885 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.911438 23702 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.994417 23832 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:18.994560 23786 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.895654 23903 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:18.995059 23755 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.995329 23903 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:18.995525 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:18.995703 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:18.995842 23903 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:18.996291 22266 raft_consensus.cc:2371] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7. Re-sending same reply.
I20250114 08:59:18.997481 23893 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:18.929343 23756 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:18.998708 23756 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:18.999847 23793 raft_consensus.cc:695] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:19.000957 23756 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:19.000994 23431 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:18.965417 21702 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
W20250114 08:59:19.007280 22241 leader_election.cc:336] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.946s (SENT)
I20250114 08:59:19.007606 21627 raft_consensus.cc:2363] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
W20250114 08:59:19.008312 22241 leader_election.cc:336] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.104s (SENT)
W20250114 08:59:19.009274 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet b3650a91ff4a4213b9091f197bf75fb9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d7732324fb5f48d3b16a29d4bb2584ed is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:19.010607 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.011816 23781 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:19.012904 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.013557 21627 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:19.014406 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:19.014593 21628 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:19.015005 21627 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:19.015726 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:19.016323 21628 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:19.017148 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 308666us
I20250114 08:59:19.020100 23899 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:18.981526 23907 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:19.020190 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 253531us
W20250114 08:59:18.990136 23901 log.cc:926] Time spent T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20250114 08:59:19.027527 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.004168 22271 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:19.006906 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.028594 21626 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:19.028594 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:19.043222 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:19.043628 21630 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:19.032557 21623 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:18.963284 23902 raft_consensus.cc:695] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:19.047076 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.051198 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.051582 21702 raft_consensus.cc:2371] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6. Re-sending same reply.
I20250114 08:59:19.052717 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.053546 21621 raft_consensus.cc:2405] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:19.026499 21628 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:19.044059 21624 raft_consensus.cc:2371] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
I20250114 08:59:19.054428 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.055160 21626 raft_consensus.cc:2388] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:19.055470 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.058210 23781 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:19.059049 21630 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 08:59:19.059095 23781 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.024581 23893 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:19.066962 21628 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20250114 08:59:19.056595 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:19.068212 21555 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:19.020865 23874 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:19.061600 21593 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:19.063131 21621 raft_consensus.cc:1212] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-4.4] Dedup: 3.3->[4.4-4.4]
I20250114 08:59:19.064385 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.070257 21549 raft_consensus.cc:2405] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:19.061240 23796 consensus_queue.cc:1046] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250114 08:59:19.071594 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.073091 23902 consensus_queue.cc:237] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.075469 21629 raft_consensus.cc:1270] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:19.076891 23796 raft_consensus.cc:3049] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:19.077286 23796 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:19.078035 23796 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.079241 23796 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:19.081166 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.081584 21450 catalog_manager.cc:5526] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 2 to 3, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:19.082599 23874 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:19.083477 23874 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.029366 23868 raft_consensus.cc:695] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:19.069475 21621 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: [3.3-4.4]
I20250114 08:59:19.068576 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.083755 21450 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 2 to 3, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:19.089350 21703 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:19.093019 21627 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:19.051666 23644 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:19.095145 21696 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250114 08:59:19.097126 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.101302 21699 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:19.104022 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.104912 21701 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:19.105302 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.105916 21697 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 7
I20250114 08:59:19.111132 22885 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:19.111735 22885 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.117427 21553 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [4.2-7.3]
I20250114 08:59:19.117165 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.117224 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.117954 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.119726 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.045655 23907 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:19.123953 21554 raft_consensus.cc:2363] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
W20250114 08:59:19.125571 21596 leader_election.cc:336] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.521s (SENT)
I20250114 08:59:19.127863 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.130362 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.120357 21696 raft_consensus.cc:2371] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7. Re-sending same reply.
I20250114 08:59:19.094029 23784 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:19.142590 21699 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:19.144143 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:19.144505 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.138343 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.149681 21702 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:19.137285 21703 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:19.138558 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.136786 22885 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:19.139382 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.156270 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.158183 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.158977 22885 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:19.143805 21450 catalog_manager.cc:5526] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:19.114956 23912 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:19.154599 21695 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:19.140805 23908 tablet_replica.cc:331] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: stopping tablet replica
I20250114 08:59:19.089428 23893 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.117561 23644 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:19.170010 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.170778 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.191010 21702 raft_consensus.cc:2363] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7.
I20250114 08:59:19.114979 23690 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
W20250114 08:59:19.178802 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 53258.8us
I20250114 08:59:19.163374 21703 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:19.181210 21698 raft_consensus.cc:1270] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:19.193042 22239 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.189640 21546 raft_consensus.cc:1235] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:19.157325 23784 raft_consensus.cc:695] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:19.206036 23844 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:59:19.208485 21450 catalog_manager.cc:5526] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:19.208611 23644 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.210980 21450 catalog_manager.cc:5526] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:19.133039 23895 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20250114 08:59:19.219206 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.161885 23915 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20250114 08:59:19.221719 23913 raft_consensus.cc:3049] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:19.222138 23913 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:19.222970 23913 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.223505 23844 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250114 08:59:19.223881 23824 consensus_meta.cc:287] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.590s user 0.006s sys 0.003s
I20250114 08:59:19.224455 23824 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.226390 23915 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:19.227015 23824 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:19.231424 22241 leader_election.cc:400] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7.
I20250114 08:59:19.231791 22241 leader_election.cc:403] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20250114 08:59:19.233958 21596 leader_election.cc:336] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.670s (SENT)
I20250114 08:59:19.235566 21555 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:19.236332 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.237455 21703 raft_consensus.cc:2405] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:19.238389 21550 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [4.2-7.3]
I20250114 08:59:19.238516 21701 raft_consensus.cc:2463] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:19.239365 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.240418 21701 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:19.241345 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.084671 23793 consensus_queue.cc:237] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:19.226923 23915 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.270641 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.224256 23913 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:19.272508 23856 consensus_queue.cc:1046] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:59:19.273702 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.274351 21701 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 2.
I20250114 08:59:19.271832 21703 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:19.242113 21702 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:19.275799 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.253094 23597 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:19.253546 23844 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:19.254323 23810 consensus_queue.cc:1035] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250114 08:59:19.182420 23912 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.283603 21697 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:19.261911 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.264745 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.284610 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.248075 23467 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7.
I20250114 08:59:19.197772 23436 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:19.267904 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.243993 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:19.276823 21703 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
W20250114 08:59:19.281351 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 122178us
I20250114 08:59:19.307030 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.307859 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.309693 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.282807 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.284744 21625 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:19.188302 23868 consensus_queue.cc:237] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.284263 21696 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:19.285194 21697 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:19.245286 21450 catalog_manager.cc:5526] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:19.226393 23903 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.321702 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.325574 21702 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
W20250114 08:59:19.325814 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 81939.7us
I20250114 08:59:19.322000 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.324220 21695 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:19.327852 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.329862 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.312356 21520 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.219273 23893 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:19.336781 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.337637 21703 raft_consensus.cc:2371] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6. Re-sending same reply.
I20250114 08:59:19.338707 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.339617 21703 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:19.305713 23537 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:19.364593 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.330695 21698 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:19.349612 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.351956 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:19.366747 21550 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:19.353930 21555 raft_consensus.cc:1270] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:19.371204 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.280781 23597 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:19.375437 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 181254us
I20250114 08:59:19.378569 21697 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:19.380049 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.381275 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.382273 21668 leader_election.cc:400] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:19.383563 21668 leader_election.cc:403] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:19.370079 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.384405 21703 raft_consensus.cc:2388] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:19.385040 21697 raft_consensus.cc:1808] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:19.385931 21697 raft_consensus.cc:2405] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:19.386722 21696 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:19.331369 23885 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:19.335086 23702 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:19.392721 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.393564 21698 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250114 08:59:19.342836 23787 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:19.389779 21596 leader_election.cc:336] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.936s (SENT)
I20250114 08:59:19.394891 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.389916 21701 raft_consensus.cc:2405] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:19.342844 23783 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:19.411854 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.412753 21625 raft_consensus.cc:2363] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:19.396584 23902 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:19.424722 21698 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:19.464111 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.408519 23856 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:19.375376 23537 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:19.389279 23784 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:19.582717 21550 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:19.597378 21629 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.014s sys 0.027s
W20250114 08:59:19.603050 21629 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.015s sys 0.027s
I20250114 08:59:19.464133 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:19.603873 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 336022us
W20250114 08:59:19.481711 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 65570.1us
I20250114 08:59:19.572547 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.605149 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.605585 21696 raft_consensus.cc:2405] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20250114 08:59:19.606307 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 228966us
W20250114 08:59:19.549230 23916 log.cc:926] Time spent T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20250114 08:59:19.608996 21700 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.018s sys 0.033s
W20250114 08:59:19.610893 21700 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.526s user 0.018s sys 0.033s
I20250114 08:59:19.608930 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.473723 23844 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:19.597141 23907 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:19.615139 21668 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:19.463631 23786 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:19.545359 23912 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:19.537747 23903 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:19.627938 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 244877us
I20250114 08:59:19.439224 23893 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.176s
I20250114 08:59:19.624260 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.473326 23597 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:19.617780 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:19.637041 23436 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:19.641198 22271 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250114 08:59:19.647315 21546 raft_consensus.cc:1235] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:19.623221 23915 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:19.654968 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.672854 21695 raft_consensus.cc:2388] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:59:19.679253 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 70325us
I20250114 08:59:19.679376 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1270) took 1010 ms. Trace:
I20250114 08:59:19.623483 23844 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:19.740979 21596 leader_election.cc:336] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.087s (SENT)
I20250114 08:59:19.679909 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.741999 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:19.769471 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 154407us
I20250114 08:59:19.688181 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.782366 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.858687 21695 raft_consensus.cc:2371] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5. Re-sending same reply.
I20250114 08:59:19.861537 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.781343 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.863278 21696 raft_consensus.cc:2405] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20250114 08:59:19.761232 23930 log.cc:926] Time spent T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.065s user 0.008s sys 0.001s
I20250114 08:59:19.866771 21553 raft_consensus.cc:1270] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250114 08:59:19.790581 23933 log.cc:926] Time spent T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20250114 08:59:19.870007 21547 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.012s sys 0.005s
I20250114 08:59:19.782622 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:19.793435 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:19.860775 23928 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20250114 08:59:19.773365 23913 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:19.772188 21698 raft_consensus.cc:2363] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:19.867830 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.797353 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.885535 21546 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 9.
I20250114 08:59:19.764354 23702 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:19.770732 21596 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:19.869997 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:19.870939 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.756531 23783 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:19.769747 23796 consensus_meta.cc:287] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.686s user 0.014s sys 0.001s
I20250114 08:59:19.759152 21630 rpcz_store.cc:276] 0114 08:59:18.669252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.060066 (+390814us) service_pool.cc:225] Handling call
0114 08:59:19.060326 (+ 260us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:19.087193 (+ 26867us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:19.117897 (+ 30704us) log.cc:844] Serialized 14 byte log entry
0114 08:59:19.383854 (+265957us) log.cc:844] Serialized 14 byte log entry
0114 08:59:19.652223 (+268369us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b18003ae820. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c60c03a 00007fa645ac8be9 00007fa63c4cd930 00007fa63c691a14 00007fa63c691305 00007fa63f5c53d0 00007fa63f5d7666 00007fa63f5e81d7 00007fa63f5e8158
0114 08:59:19.678911 (+ 26688us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:19.678963 (+ 52us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:19.679002 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:19.679075 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:19.679313 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26031,"prepare.queue_time_us":33548,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":4439,"raft.queue_time_us":294988,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":12453,"spinlock_wait_cycles":31011456,"thread_start_us":243995,"threads_started":2,"wal-append.queue_time_us":419446}
W20250114 08:59:19.873585 21547 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.013s sys 0.005s
I20250114 08:59:19.924034 21550 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:19.781654 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:19.772809 23907 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:19.876020 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:19.865849 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:19.928738 21627 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 9.
I20250114 08:59:19.791992 23597 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:19.800057 23537 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:19.797044 23844 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:19.782073 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:19.930047 22240 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:19.874853 23874 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.006s sys 0.020s
I20250114 08:59:19.931411 21699 raft_consensus.cc:2388] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:19.931874 21668 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:19.919605 21555 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:19.764964 23472 consensus_queue.cc:1035] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.112s
I20250114 08:59:19.925215 21704 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:19.759024 23781 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.125s
I20250114 08:59:19.764914 23935 consensus_queue.cc:1046] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.145s
I20250114 08:59:19.882181 23903 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.056s
I20250114 08:59:19.883767 23913 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:19.935266 23874 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.007s sys 0.021s
W20250114 08:59:19.948580 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 166038us
I20250114 08:59:19.931439 23937 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:19.930918 23868 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:19.943447 23862 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:19.940615 23907 raft_consensus.cc:695] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:19.942950 23893 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:19.972647 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 231689us
I20250114 08:59:19.963317 21555 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:19.968022 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:20.003535 21555 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [4.2-7.3]
I20250114 08:59:20.007747 21697 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:20.015794 23942 log.cc:926] Time spent T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250114 08:59:20.017302 23923 log.cc:926] Time spent T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250114 08:59:20.019086 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1482) took 1031 ms. Trace:
I20250114 08:59:20.019444 21621 rpcz_store.cc:276] 0114 08:59:18.987993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.093174 (+105181us) service_pool.cc:225] Handling call
0114 08:59:19.093457 (+ 283us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:19.093653 (+ 196us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:19.093669 (+ 16us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:19.108428 (+ 14759us) log.cc:844] Serialized 26 byte log entry
0114 08:59:19.481925 (+373497us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:19.532258 (+ 50333us) log.cc:844] Serialized 14 byte log entry
0114 08:59:19.758712 (+226454us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:19.758800 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:20.018851 (+260051us) raft_consensus.cc:1680] finished
0114 08:59:20.018891 (+ 40us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:20.019022 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":231036,"prepare.queue_time_us":52161,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":227462,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":64,"thread_start_us":299167,"threads_started":1,"wal-append.queue_time_us":742002}
I20250114 08:59:19.967255 23783 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:20.024664 21596 leader_election.cc:336] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.118s (SENT)
I20250114 08:59:20.025588 21596 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.025918 23908 raft_consensus.cc:2238] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 08:59:20.026954 23908 raft_consensus.cc:2267] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 08:59:20.030066 21699 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:20.030352 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1489) took 1040 ms. Trace:
I20250114 08:59:20.030834 21623 rpcz_store.cc:276] 0114 08:59:18.990246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.055232 (+ 64986us) service_pool.cc:225] Handling call
0114 08:59:20.030141 (+974909us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b540039e754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:20.030294 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2052192000}
I20250114 08:59:20.059862 21628 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:20.061039 21669 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.059273 23937 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:20.063540 21555 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:59:19.966109 23862 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:20.064956 21700 raft_consensus.cc:1270] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 08:59:20.064414 21704 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:20.065274 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1479) took 1182 ms. Trace:
W20250114 08:59:20.059356 23927 log.cc:926] Time spent T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20250114 08:59:20.068439 21629 rpcz_store.cc:276] 0114 08:59:18.882434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.076231 (+193797us) service_pool.cc:225] Handling call
0114 08:59:19.076598 (+ 367us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:19.084237 (+ 7639us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:19.084272 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:19.084819 (+ 547us) log.cc:844] Serialized 52 byte log entry
0114 08:59:19.603695 (+518876us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:19.613293 (+ 9598us) log.cc:844] Serialized 14 byte log entry
0114 08:59:19.759082 (+145789us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:19.759154 (+ 72us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:20.021329 (+262175us) raft_consensus.cc:1680] finished
0114 08:59:20.021351 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:20.065197 (+ 43846us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":145483,"prepare.queue_time_us":9202,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":1841,"raft.queue_time_us":153627,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158,"thread_start_us":518467,"threads_started":1,"wal-append.queue_time_us":575734}
I20250114 08:59:20.073359 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:20.074374 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 72135.6us
W20250114 08:59:20.075385 21596 leader_election.cc:336] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.120s (SENT)
I20250114 08:59:20.076308 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.074394 23907 consensus_queue.cc:237] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.063527 23796 consensus_queue.cc:1046] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20250114 08:59:20.078197 23868 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:20.078236 23950 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250114 08:59:20.079046 23796 raft_consensus.cc:3049] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:20.079455 23796 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:20.068791 23862 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:20.068025 23893 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.085616 21696 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250114 08:59:20.090106 21596 leader_election.cc:336] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 2.404s (SENT)
I20250114 08:59:20.080289 23796 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.090624 21521 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.095686 21596 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:20.086974 23924 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.159s user 0.001s sys 0.003s
I20250114 08:59:20.093364 21699 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:20.096549 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.097977 23796 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:20.098649 23893 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:20.030979 23913 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.102487 21669 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.105276 21521 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:20.111105 23908 ts_tablet_manager.cc:1905] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250114 08:59:20.111867 21596 connection.cc:232] RPC call timeout handler was delayed by -2.0007714509999914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20250114 08:59:20.112435 21596 connection.cc:232] RPC call timeout handler was delayed by -1.1304571339999256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20250114 08:59:20.113093 23792 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:20.113664 23792 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.116782 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1472) took 1491 ms. Trace:
I20250114 08:59:20.116902 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1471) took 1541 ms. Trace:
I20250114 08:59:20.117182 21622 rpcz_store.cc:276] 0114 08:59:18.625498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:18.930261 (+304763us) service_pool.cc:225] Handling call
0114 08:59:20.116576 (+1186315us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54003b9cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:20.116703 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2307396992}
I20250114 08:59:20.117336 21624 rpcz_store.cc:276] 0114 08:59:18.576029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.064809 (+488780us) service_pool.cc:225] Handling call
0114 08:59:19.095532 (+ 30723us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:19.106045 (+ 10513us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:19.106091 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:19.140990 (+ 34899us) log.cc:844] Serialized 26 byte log entry
0114 08:59:19.600843 (+459853us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:19.600912 (+ 69us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:19.601024 (+ 112us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:20.063526 (+462502us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:20.116571 (+ 53045us) raft_consensus.cc:1680] finished
0114 08:59:20.116593 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:20.116835 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14791,"raft.queue_time_us":10808,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":11042,"spinlock_wait_cycles":10062208,"thread_start_us":332659,"threads_started":1,"wal-append.queue_time_us":707461}
W20250114 08:59:20.119208 23921 log.cc:926] Time spent T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.364s user 0.000s sys 0.001s
I20250114 08:59:20.119058 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.119733 21550 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:20.121060 23907 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:20.121472 23907 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.121856 23907 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:20.122787 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 199) took 1495 ms. Trace:
I20250114 08:59:20.100664 23537 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:20.125276 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:20.104177 23937 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:20.128558 23781 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:20.128659 21594 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:20.129272 23781 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:20.129943 23937 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:20.130462 23937 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:20.130271 23781 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.130908 23937 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:20.134092 23913 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:20.123188 21626 rpcz_store.cc:276] 0114 08:59:18.627771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.063659 (+435888us) service_pool.cc:225] Handling call
0114 08:59:19.064055 (+ 396us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:19.073009 (+ 8954us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:19.073039 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:19.115293 (+ 42254us) log.cc:844] Serialized 26 byte log entry
0114 08:59:19.448558 (+333265us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:19.448592 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:19.448675 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:20.118651 (+669976us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:20.122373 (+ 3722us) raft_consensus.cc:1680] finished
0114 08:59:20.122394 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:20.122720 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8696,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":47890176,"thread_start_us":292493,"threads_started":1,"wal-append.queue_time_us":491853}
W20250114 08:59:20.160624 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 55440.8us
I20250114 08:59:20.158804 21669 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.161974 21521 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:20.181181 21669 leader_election.cc:400] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:20.181646 21669 leader_election.cc:403] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:20.182384 23868 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:20.183854 21669 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.184578 23868 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:20.185751 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.186049 23690 raft_consensus.cc:2798] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:20.186648 23690 raft_consensus.cc:695] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:20.187539 23868 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:20.187674 23690 consensus_queue.cc:237] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.189345 21668 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.190896 21669 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.186591 21521 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.193707 21521 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.193964 23903 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:20.194540 23903 raft_consensus.cc:695] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:20.195618 23903 consensus_queue.cc:237] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.196870 23899 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:20.197413 23899 raft_consensus.cc:695] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:20.198335 23899 consensus_queue.cc:237] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.115988 23792 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.193562 23597 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:20.205055 23597 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:20.207885 23597 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:20.209684 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:20.160185 23537 raft_consensus.cc:695] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:20.104135 23801 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:20.201330 21521 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.177997 23787 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:20.216710 23787 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:20.217130 23787 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:20.203907 23786 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:20.217629 23786 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.219671 23786 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.220427 23874 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 08:59:20.220963 23874 raft_consensus.cc:695] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:20.221463 22270 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:59:20.221901 23874 consensus_queue.cc:237] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.222079 23862 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.213392 23783 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250114 08:59:20.225394 23936 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:20.226083 23783 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:20.212388 23824 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:20.216126 21521 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:20.229437 21521 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:20.230067 21521 raft_consensus.cc:2681] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 08:59:20.231089 23702 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:20.231197 23467 raft_consensus.cc:3049] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:20.231607 23467 raft_consensus.cc:738] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:20.232496 23467 consensus_queue.cc:260] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.202589 23472 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 08:59:20.220294 21520 leader_election.cc:336] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.539s (SENT)
W20250114 08:59:20.239485 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 48662.1us
I20250114 08:59:20.249042 21669 leader_election.cc:400] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 2. Current term is 3.
I20250114 08:59:20.245994 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.258219 21626 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:20.233731 23467 raft_consensus.cc:3054] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:20.266182 21596 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.268631 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.158543 23893 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.275231 21626 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 10.
I20250114 08:59:20.268684 23862 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.282249 23431 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:20.284044 21596 leader_election.cc:400] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9.
I20250114 08:59:20.307631 21696 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:20.309460 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.312145 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.315573 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.316275 21630 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:20.316604 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.317566 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.277637 23793 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.235968 23597 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.326493 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.326830 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.328126 21624 raft_consensus.cc:2371] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
I20250114 08:59:20.307958 23431 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:20.306804 23822 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:20.339854 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 90937.4us
I20250114 08:59:20.340731 21669 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.342598 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.343349 21626 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:20.344601 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.345284 21622 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:20.346908 21621 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:20.248384 23801 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.354339 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.359617 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.358168 21630 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:20.360239 21624 raft_consensus.cc:2371] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7. Re-sending same reply.
I20250114 08:59:20.363346 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.363992 21623 raft_consensus.cc:2371] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4. Re-sending same reply.
W20250114 08:59:20.365689 21439 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet b3650a91ff4a4213b9091f197bf75fb9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250114 08:59:20.338327 23702 raft_consensus.cc:695] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:20.373652 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.377558 23597 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.378966 23893 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:20.380213 23955 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20250114 08:59:20.348074 23822 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.343164 23537 consensus_queue.cc:237] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.343564 23431 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.361614 23913 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.384506 23801 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.385313 21669 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.458940 21629 raft_consensus.cc:2405] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 08:59:20.353614 22270 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:20.399758 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1297) took 1519 ms. Trace:
I20250114 08:59:20.459563 21628 rpcz_store.cc:276] 0114 08:59:18.879905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.067983 (+188078us) service_pool.cc:225] Handling call
0114 08:59:20.031131 (+963148us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:20.066956 (+ 35825us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:20.399468 (+332512us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b540039e750. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:20.399670 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2149725440}
I20250114 08:59:20.412778 23787 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.451061 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:20.466617 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 81372.4us
I20250114 08:59:20.472131 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.474573 21628 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:20.477931 21669 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.483484 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.484546 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.484778 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.483615 21669 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.505972 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.506940 21625 raft_consensus.cc:2405] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 08:59:20.507663 21669 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.509341 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.475867 23893 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:20.513501 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.482136 23824 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:20.481817 23472 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:20.511684 21624 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:20.525004 23973 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.529479 21669 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.510571 23915 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.511287 23913 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.471369 23907 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:20.548982 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:20.540493 23962 log.cc:926] Time spent T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20250114 08:59:20.554471 21624 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:20.554581 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.539487 23824 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:20.518321 23936 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250114 08:59:20.557930 21669 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.561183 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.514228 23702 consensus_queue.cc:237] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.553961 23903 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:20.555531 23912 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:20.569443 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.573038 21552 raft_consensus.cc:2405] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250114 08:59:20.546777 23937 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.559603 23913 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:20.562667 23824 raft_consensus.cc:695] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:20.594143 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 80712.8us
I20250114 08:59:20.571702 23903 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.596940 23792 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.600798 23787 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:20.601318 23970 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250114 08:59:20.597728 23903 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:20.598444 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.597893 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.608361 21669 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.581774 23822 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.610147 21669 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.613459 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:20.614763 23965 log.cc:926] Time spent T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20250114 08:59:20.596102 23431 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:20.616420 23824 consensus_queue.cc:237] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.609025 23913 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:20.623984 21669 leader_election.cc:400] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
I20250114 08:59:20.624496 23907 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:20.624581 23472 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 08:59:20.625187 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:20.620594 23893 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:20.627437 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:20.628062 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:20.629531 23959 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20250114 08:59:20.629987 23964 log.cc:926] Time spent T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20250114 08:59:20.612531 23801 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:20.610658 23915 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.610499 23912 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:20.601351 23792 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.653951 23792 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.654832 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.654723 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.626056 23937 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.637394 23893 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.634115 21669 leader_election.cc:400] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
I20250114 08:59:20.660528 21520 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:20.661448 21669 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.664544 21520 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.664206 23912 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:20.666432 23912 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:20.648916 23966 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:20.671386 23787 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:20.671841 23787 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:20.672184 23787 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:20.672801 23967 log.cc:926] Time spent T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.167s user 0.004s sys 0.000s
I20250114 08:59:20.663506 23893 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:20.675915 23903 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.678035 23903 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:20.690265 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:20.691682 22266 raft_consensus.cc:2405] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250114 08:59:20.708503 21555 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:20.665025 23793 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:20.696080 21596 leader_election.cc:336] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485): Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.703s (SENT)
I20250114 08:59:20.664312 23899 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.721534 21669 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.723877 21593 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:20.725816 21669 leader_election.cc:400] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
W20250114 08:59:20.722965 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 58496.7us
I20250114 08:59:20.726660 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.728109 21669 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.667644 23912 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.729344 23913 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:20.739952 23913 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:20.741305 22266 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250114 08:59:20.727748 23894 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.743525 23973 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 08:59:20.729374 23936 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.019s
I20250114 08:59:20.710041 23903 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:20.722343 23793 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:20.739089 23986 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250114 08:59:20.764031 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.758991 23936 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.725432 23781 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250114 08:59:20.779717 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 51661.4us
I20250114 08:59:20.780611 21669 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.719651 23787 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.782476 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.730317 23792 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.725575 23907 consensus_queue.cc:1035] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20250114 08:59:20.789623 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.772087 23973 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:20.790510 21626 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:20.797633 21521 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.002s sys 0.015s
W20250114 08:59:20.798106 21521 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.003s sys 0.015s
W20250114 08:59:20.798429 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 562694us
I20250114 08:59:20.802527 21596 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.826113 21596 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:20.832968 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.833963 21626 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:20.751281 23893 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:20.777824 23781 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:20.850072 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:20.850893 21555 raft_consensus.cc:604] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:20.851361 21555 raft_consensus.cc:612] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:20.784085 23894 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.860769 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.859454 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.860191 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.861899 21628 raft_consensus.cc:1270] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:20.860903 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:20.862416 21623 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:20.791947 23902 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:20.793953 23907 consensus_queue.cc:1225] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:20.857038 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.873194 21626 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:20.862049 21622 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:20.878048 21624 raft_consensus.cc:1270] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:20.765170 23903 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.880208 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:20.884965 21450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55864 (request call id 192) took 23846 ms (client timeout 15000 ms). Trace:
I20250114 08:59:20.885912 21627 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:20.854687 23792 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:20.890318 23792 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.890784 23792 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:20.774739 23936 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.898874 23936 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:20.900239 23993 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.901016 23993 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:20.885454 21450 rpcz_store.cc:269] 0114 08:58:57.038233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:58:57.039255 (+ 1022us) service_pool.cc:225] Handling call
0114 08:59:20.884875 (+23845620us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:19.710940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:19.803754 (+ 92814us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:19.803805 (+ 51us) write_op.cc:435] Acquired schema lock
0114 08:59:19.803830 (+ 25us) tablet.cc:659] Decoding operations
0114 08:59:19.805215 (+ 1385us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:19.805292 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:19.805332 (+ 40us) tablet.cc:682] Acquiring locks for 6 operations
0114 08:59:19.805996 (+ 664us) tablet.cc:698] Row locks acquired
0114 08:59:19.806019 (+ 23us) write_op.cc:260] PREPARE: finished
0114 08:59:19.806212 (+ 193us) write_op.cc:270] Start()
0114 08:59:19.806357 (+ 145us) write_op.cc:276] Timestamp: P: 1736845159806154 usec, L: 0
0114 08:59:19.806380 (+ 23us) op_driver.cc:348] REPLICATION: starting
0114 08:59:19.921252 (+114872us) log.cc:844] Serialized 2494 byte log entry
0114 08:59:20.279002 (+357750us) op_driver.cc:464] REPLICATION: finished
0114 08:59:20.524511 (+245509us) write_op.cc:301] APPLY: starting
0114 08:59:20.524603 (+ 92us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:20.524794 (+ 191us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:20.524923 (+ 129us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:20.563322 (+ 38399us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:20.580487 (+ 17165us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0114 08:59:20.593159 (+ 12672us) write_op.cc:312] APPLY: finished
0114 08:59:20.621644 (+ 28485us) log.cc:844] Serialized 52 byte log entry
0114 08:59:20.630719 (+ 9075us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:20.630821 (+ 102us) write_op.cc:454] Released schema lock
0114 08:59:20.630855 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:20.631183 (+ 328us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:20.631211 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":230023,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":231750,"apply.run_cpu_time_us":3482,"apply.run_wall_time_us":110328,"num_ops":6,"prepare.queue_time_us":201725,"prepare.run_cpu_time_us":53021,"prepare.run_wall_time_us":361110,"raft.queue_time_us":179480,"raft.run_cpu_time_us":26260,"raft.run_wall_time_us":145078,"replication_time_us":472460,"thread_start_us":481029,"threads_started":4,"wal-append.queue_time_us":166518}]]}
I20250114 08:59:20.919016 21622 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:20.850215 23824 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.854081 23781 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:20.921200 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 122065us
I20250114 08:59:20.926162 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:20.933216 21626 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 08:59:20.865399 23937 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:20.935688 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.937566 21626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:55318
W20250114 08:59:20.938313 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 81398.2us
I20250114 08:59:20.939438 21596 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.938426 21626 raft_consensus.cc:604] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.941985 21626 raft_consensus.cc:612] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:20.946146 21596 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.948197 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.946213 21555 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:20.948587 23783 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.950317 23993 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:20.951090 23783 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.954735 21596 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.955801 23431 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:20.957837 23537 consensus_queue.cc:1035] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:20.960495 23793 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.964423 23787 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:20.933578 23796 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:20.966964 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.937307 23937 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:20.969610 21596 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.973366 23903 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:20.970445 23786 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:20.943931 23936 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:20.982508 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:20.987387 21555 raft_consensus.cc:604] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.987876 21555 raft_consensus.cc:612] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:20.944778 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:20.951112 23893 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:20.994367 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:20.940830 23824 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:20.977826 23937 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:20.996248 23903 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.006639 21596 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:20.978385 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:21.024681 21630 raft_consensus.cc:1270] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 08:59:21.025214 21630 pending_rounds.cc:77] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 1
I20250114 08:59:21.025514 21630 pending_rounds.cc:99] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:21.025837 21630 raft_consensus.cc:2883] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:21.011124 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.011373 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.012892 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.026932 21697 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:21.027272 21704 raft_consensus.cc:2405] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20250114 08:59:21.017356 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 63502.7us
W20250114 08:59:21.019245 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 226368us
I20250114 08:59:20.954794 23984 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20250114 08:59:20.977365 23908 ts_tablet_manager.cc:1918] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20250114 08:59:21.008030 23936 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:21.037199 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.026579 21596 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.017792 23472 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.017606 23893 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.044867 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.017449 23913 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.000931 23784 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 08:59:21.046454 21521 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.079154 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:21.079494 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 53027.8us
I20250114 08:59:21.073348 21697 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 10.
I20250114 08:59:21.044832 21593 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:21.080969 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:21.081449 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.041461 23793 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:21.117471 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 71122us
I20250114 08:59:21.118674 21697 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250114 08:59:21.119545 21697 pending_rounds.cc:77] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 2
I20250114 08:59:21.119885 21697 pending_rounds.cc:99] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 7.3
I20250114 08:59:21.018240 23903 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:21.044973 23908 log.cc:1198] T 3b369b35736c413ca069db061ed36bc5 P 28a83b7a7d9d4aa899b425086aa45a8e: Deleting WAL directory at /tmp/dist-test-taskD6FxqY/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736845029639699-20382-0/minicluster-data/ts-0-root/wals/3b369b35736c413ca069db061ed36bc5
I20250114 08:59:21.124207 21555 raft_consensus.cc:604] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:21.127025 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 82249.8us
I20250114 08:59:21.117522 21596 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.129534 21697 raft_consensus.cc:2883] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:21.162240 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.163414 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.132009 21704 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:21.080094 23431 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.130338 23796 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:21.085492 23893 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.081065 23966 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:21.141978 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.143728 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.149538 24004 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:21.177675 21703 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:21.152405 23702 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:21.125010 23899 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:21.188374 23899 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:21.189260 23899 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:21.190937 22241 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:21.191984 23862 raft_consensus.cc:2798] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:21.192561 23862 raft_consensus.cc:695] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [term 10 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:21.193827 23702 raft_consensus.cc:695] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:21.193666 23862 consensus_queue.cc:237] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:21.194772 23702 consensus_queue.cc:237] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.195724 21669 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:21.196547 23997 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:21.196916 23997 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:21.197264 23997 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:21.117214 23913 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.199954 23913 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.201732 23824 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:21.202394 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.202852 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.203032 21700 raft_consensus.cc:2371] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 10. Re-sending same reply.
I20250114 08:59:21.203701 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:21.203946 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.204049 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.204800 21703 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:21.172009 23783 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:21.206184 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.206621 23824 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:21.080505 23472 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:21.208480 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.209309 23824 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:21.209724 23472 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:21.210863 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:21.211055 21669 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.211897 23893 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.212708 23824 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:59:21.213624 21696 raft_consensus.cc:1235] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250114 08:59:21.214071 23824 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.212107 23997 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.154793 23792 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.155364 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.078975 23874 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
W20250114 08:59:21.087947 23991 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
W20250114 08:59:21.165177 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 47755us
I20250114 08:59:21.164458 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.251538 21704 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:21.175254 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:21.152279 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 73244us
I20250114 08:59:21.262650 21697 raft_consensus.cc:2405] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:21.264056 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.219848 21700 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:21.249665 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.090523 23597 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20250114 08:59:21.207281 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.272589 21701 raft_consensus.cc:2371] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3. Re-sending same reply.
I20250114 08:59:21.279176 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.280773 21703 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:21.274147 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.281623 21697 raft_consensus.cc:2371] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5. Re-sending same reply.
I20250114 08:59:21.282851 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.284296 21697 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:21.278719 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.286059 21696 raft_consensus.cc:2405] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250114 08:59:21.287587 21596 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.289366 21596 leader_election.cc:400] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:21.289795 21596 leader_election.cc:403] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 08:59:21.305074 21701 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:21.308372 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:21.309340 21697 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250114 08:59:21.229358 23824 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.318370 21668 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.315766 21698 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
W20250114 08:59:21.339663 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 50473.7us
I20250114 08:59:21.340962 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.352928 21596 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.346338 21697 raft_consensus.cc:1270] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250114 08:59:21.257915 23913 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.103s
I20250114 08:59:21.358773 23907 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.361689 21698 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:21.359001 23786 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:21.360304 23792 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:21.357702 21596 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.366540 23936 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:21.365352 21704 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:21.368896 21698 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:21.370229 21596 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.336848 23783 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:21.357091 23824 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.376405 23932 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:21.377995 23903 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:21.378700 23999 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.379460 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:21.228026 23893 consensus_queue.cc:1035] T 9690a0a2cde84c5d9312c8694185a0b2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20250114 08:59:21.337249 23796 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.380033 23912 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.379860 23784 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.380594 23997 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.304543 23781 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.398442 23786 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.408622 23908 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33868 (request call id 270) took 2690 ms. Trace:
I20250114 08:59:21.415910 21451 catalog_manager.cc:5526] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 7, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 7 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:21.426041 21550 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:21.427512 21697 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:21.305732 23915 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.432319 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:21.432991 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 62815.6us
I20250114 08:59:21.433013 21550 raft_consensus.cc:3054] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:21.435778 23781 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:21.407568 23999 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:21.433211 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 274194us
I20250114 08:59:21.433223 21451 catalog_manager.cc:5526] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 6 to 7, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 7 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:21.416339 23937 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.450507 21451 catalog_manager.cc:5526] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:21.444496 21521 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.461809 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:21.470410 21596 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:21.471689 21630 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20250114 08:59:21.436667 23786 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.472867 21451 catalog_manager.cc:5526] T cc9f4dcde58e4d508d5ffbbf301a7a6b P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 7, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 7 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:21.476147 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:21.476974 21629 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:21.476802 21451 catalog_manager.cc:5526] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 5, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 5 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:21.478399 21550 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 10.
I20250114 08:59:21.478758 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:21.430982 23912 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.482163 23894 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.446229 23935 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:21.473738 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:21.476558 23936 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.012s
I20250114 08:59:21.483008 23431 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250114 08:59:21.478484 21451 catalog_manager.cc:5526] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 2 to 5, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:21.500094 23936 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:21.432691 23915 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.507315 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:21.506461 21451 catalog_manager.cc:5526] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 2 to 3, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:21.495716 21629 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:21.512993 21630 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:21.514618 21451 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 2 to 3, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 3 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:21.458029 23932 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20250114 08:59:21.520941 21626 raft_consensus.cc:1270] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:21.526712 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:21.527695 21622 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:21.491214 23690 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:21.492668 23935 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:21.486850 23912 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.537882 21521 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:21.507637 23793 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:21.425802 23908 rpcz_store.cc:276] 0114 08:59:18.703933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:18.707447 (+ 3514us) service_pool.cc:225] Handling call
0114 08:59:19.096738 (+389291us) ts_tablet_manager.cc:1145] Deleting tablet 3b369b35736c413ca069db061ed36bc5
0114 08:59:19.096848 (+ 110us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 08:59:20.628224 (+1531376us) tablet_metadata.cc:641] Metadata flushed
0114 08:59:20.968674 (+340450us) tablet_metadata.cc:641] Metadata flushed
0114 08:59:21.389097 (+420423us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":649043,"tablet-delete.queue_time_us":316188,"thread_start_us":189401,"threads_started":1}
I20250114 08:59:21.534801 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:21.512279 23936 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:21.483697 23899 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.486801 23786 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.482728 23894 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.561964 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:21.582074 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:21.582854 21625 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:21.546136 23793 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:21.594108 21669 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:21.595214 21547 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:21.595659 21669 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:21.596700 23903 consensus_queue.cc:1046] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:21.597658 23903 consensus_queue.cc:1225] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:21.598285 23903 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:21.599061 23903 raft_consensus.cc:2798] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:21.599629 23903 raft_consensus.cc:695] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:21.601038 23868 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:21.601763 23868 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.606751 21669 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:21.604123 23868 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.609097 23868 consensus_queue.cc:1046] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:21.610550 23903 consensus_queue.cc:237] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.555967 23936 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:21.550280 23824 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.564285 23914 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:21.557644 24004 consensus_queue.cc:1035] T f466e3efd6cb41329852e9028490bdf7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20250114 08:59:21.615528 23793 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:21.662689 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:21.639506 23824 raft_consensus.cc:3049] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:21.717478 21555 raft_consensus.cc:1270] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:21.718561 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:21.721078 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:21.683853 21695 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1aeae7c29264f839189001364c51746"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
mode: GRACEFUL
new_leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
from {username='slave'} at 127.0.0.1:54510
W20250114 08:59:21.688284 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 217973us
I20250114 08:59:21.722846 21695 raft_consensus.cc:604] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Received request to transfer leadership to TS 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:21.723259 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:21.723229 21695 raft_consensus.cc:612] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:21.732138 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:21.663214 23914 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:21.742117 21555 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 08:59:21.743152 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:21.756500 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 173060us
I20250114 08:59:21.695302 23937 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:21.695140 23824 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:21.761708 24006 log.cc:926] Time spent T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250114 08:59:21.726243 23690 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:21.750844 23913 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:21.724637 23999 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:21.786995 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:21.815155 21624 raft_consensus.cc:2405] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 08:59:21.805131 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:21.829418 21695 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 08:59:21.833629 21551 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:21.835750 21555 raft_consensus.cc:1270] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:21.846881 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:21.847252 21547 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:21.877583 23903 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:21.855370 22270 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:21.860872 23899 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:21.834582 23690 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.880075 23786 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:21.883291 23907 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:21.900688 24008 log.cc:926] Time spent T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20250114 08:59:21.886768 23894 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:21.950457 21696 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.004s sys 0.007s
W20250114 08:59:21.953612 21696 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s user 0.005s sys 0.007s
I20250114 08:59:21.861902 23824 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:21.927412 23786 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:21.929811 24032 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:21.880365 23914 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:21.929090 23907 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:22.000618 23786 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:22.014631 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:22.015008 23907 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:21.908231 23903 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:22.016327 23786 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:22.004627 24027 log.cc:926] Time spent T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
W20250114 08:59:22.005270 24020 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.162s user 0.000s sys 0.001s
W20250114 08:59:22.005494 24018 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
W20250114 08:59:22.006309 24019 log.cc:926] Time spent T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20250114 08:59:22.006997 23690 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:22.020844 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:22.032507 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 424142us
I20250114 08:59:22.119556 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:22.058189 21547 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250114 08:59:22.061502 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:22.131588 21698 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:22.121217 21695 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
W20250114 08:59:22.152673 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 347098us
I20250114 08:59:22.171609 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1707) took 1039 ms. Trace:
I20250114 08:59:22.172072 21697 rpcz_store.cc:276] 0114 08:59:21.131745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.356562 (+224817us) service_pool.cc:225] Handling call
0114 08:59:21.360609 (+ 4047us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:21.428075 (+ 67466us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:22.171314 (+743239us) spinlock_profiling.cc:243] Waited 66.4 ms on lock 0x7b5400327b54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:22.171511 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152762624}
I20250114 08:59:22.173719 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:22.017278 23903 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:22.216023 21518 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:22.215332 23899 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:22.283144 23899 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:22.283500 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:22.285403 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:22.286641 21629 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250114 08:59:22.286816 21438 catalog_manager.cc:4872] TS 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): tablet 3b369b35736c413ca069db061ed36bc5 (table test-workload [id=c01c36aed5b24598bdaa599e3312866e]) successfully deleted
I20250114 08:59:22.287304 23781 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:22.287719 23781 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:22.288074 23781 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:22.288504 23899 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:22.288847 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:22.289386 21546 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:22.290562 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 115761us
I20250114 08:59:22.291119 21546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0b4ce684a6943809542800e14d94079"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:22.291676 21546 raft_consensus.cc:604] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:22.292003 21546 raft_consensus.cc:612] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250114 08:59:22.292161 21520 leader_election.cc:336] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.674s (SENT)
I20250114 08:59:22.292670 24015 consensus_queue.cc:1046] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:22.293275 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250114 08:59:22.295730 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:22.296651 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:22.366175 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:22.370996 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 197927us
I20250114 08:59:22.418927 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1345) took 1339 ms. Trace:
W20250114 08:59:22.419142 21593 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.028s sys 0.031s
W20250114 08:59:22.424557 21593 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.028s sys 0.031s
I20250114 08:59:22.430245 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:22.431097 21695 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:22.419509 21701 rpcz_store.cc:276] 0114 08:59:21.079964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.306110 (+226146us) service_pool.cc:225] Handling call
0114 08:59:21.306498 (+ 388us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.306941 (+ 443us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.306981 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.336310 (+ 29329us) log.cc:844] Serialized 26 byte log entry
0114 08:59:21.649451 (+313141us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:22.099132 (+449681us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.099528 (+ 396us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:22.099641 (+ 113us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:22.418624 (+318983us) raft_consensus.cc:1680] finished
0114 08:59:22.418656 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:22.418838 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":220953,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":4671,"raft.queue_time_us":32714,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":48063,"thread_start_us":651475,"threads_started":2,"wal-append.queue_time_us":415012}
I20250114 08:59:22.457427 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:22.638797 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:22.640882 22237 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.036s sys 0.161s
I20250114 08:59:22.215122 23467 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.426s
W20250114 08:59:22.641294 22237 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.832s user 0.036s sys 0.161s
I20250114 08:59:22.661779 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1344) took 1622 ms. Trace:
I20250114 08:59:22.460186 21701 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:22.662254 21703 rpcz_store.cc:276] 0114 08:59:21.039464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.304718 (+265254us) service_pool.cc:225] Handling call
0114 08:59:21.305115 (+ 397us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.355553 (+ 50438us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.355583 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.384015 (+ 28432us) log.cc:844] Serialized 26 byte log entry
0114 08:59:21.617157 (+233142us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:21.827945 (+210788us) log.cc:844] Serialized 14 byte log entry
0114 08:59:21.828247 (+ 302us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:21.828345 (+ 98us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:22.661525 (+833180us) raft_consensus.cc:1680] finished
0114 08:59:22.661549 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:22.661701 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17091,"prepare.queue_time_us":243277,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":119435,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":20668,"thread_start_us":356868,"threads_started":2,"wal-append.queue_time_us":334619}
I20250114 08:59:22.686833 21701 raft_consensus.cc:1270] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250114 08:59:22.689077 21701 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:22.716178 21695 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
W20250114 08:59:22.731572 21594 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.967s user 0.166s sys 0.258s
I20250114 08:59:22.734520 21697 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:22.738226 21594 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.974s user 0.166s sys 0.258s
I20250114 08:59:22.744354 21698 raft_consensus.cc:1270] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 2. Preceding OpId from leader: term: 8 index: 3. (index mismatch)
W20250114 08:59:22.738554 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 974499us
I20250114 08:59:22.752074 21669 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:22.751443 21627 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.027s sys 0.070s
W20250114 08:59:22.754196 21627 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s user 0.027s sys 0.070s
I20250114 08:59:22.658360 23950 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:22.660710 24025 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:22.814743 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 143) took 1804 ms. Trace:
W20250114 08:59:22.811271 21704 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.054s user 0.043s sys 0.054s
I20250114 08:59:22.663780 23786 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:22.830639 21704 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.073s user 0.043s sys 0.054s
I20250114 08:59:22.722949 23796 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:22.642268 23907 consensus_queue.cc:1035] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.174s
W20250114 08:59:22.813815 23914 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.034s sys 0.039s
W20250114 08:59:22.860770 22241 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.043s sys 0.035s
I20250114 08:59:22.830412 21702 rpcz_store.cc:276] 0114 08:59:21.009927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.010060 (+ 133us) service_pool.cc:225] Handling call
0114 08:59:21.283635 (+273575us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b5800297398. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656d0e 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:21.283797 (+ 162us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:21.310138 (+ 26341us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.310184 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:21.355185 (+ 45001us) log.cc:844] Serialized 52 byte log entry
0114 08:59:21.660930 (+305745us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:22.137397 (+476467us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.138285 (+ 888us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.138571 (+ 286us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:22.138669 (+ 98us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:22.814415 (+675746us) raft_consensus.cc:1680] finished
0114 08:59:22.814444 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:22.814660 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33387,"prepare.queue_time_us":225804,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":94,"raft.queue_time_us":15590,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":22960,"spinlock_wait_cycles":560104960,"thread_start_us":656949,"threads_started":2,"wal-append.queue_time_us":477511}
W20250114 08:59:22.849915 22240 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.008s sys 0.027s
W20250114 08:59:22.925884 22239 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.004s sys 0.006s
I20250114 08:59:22.936178 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:22.936524 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:22.937314 22241 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.044s sys 0.037s
I20250114 08:59:22.942164 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:22.937280 22240 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.720s user 0.013s sys 0.028s
W20250114 08:59:22.938131 22239 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.005s sys 0.006s
W20250114 08:59:22.953126 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 735642us
I20250114 08:59:22.953671 21549 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
W20250114 08:59:22.849406 23784 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.059s user 0.036s sys 0.073s
W20250114 08:59:22.862620 24018 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
W20250114 08:59:22.937280 23914 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s user 0.038s sys 0.039s
I20250114 08:59:22.849542 24030 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:22.964892 24020 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
W20250114 08:59:23.000871 21521 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.002s sys 0.011s
I20250114 08:59:22.927958 24039 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:22.965543 23784 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.188s user 0.040s sys 0.092s
W20250114 08:59:23.023207 21521 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.002s sys 0.012s
I20250114 08:59:22.972029 23472 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:22.971125 24050 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:22.987466 23997 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:23.016578 24019 log.cc:926] Time spent T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
W20250114 08:59:23.054840 21550 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.019s sys 0.039s
W20250114 08:59:23.089448 21550 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.019s sys 0.039s
W20250114 08:59:23.016311 24028 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250114 08:59:22.848155 23950 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:23.064203 21596 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.333s user 0.019s sys 0.031s
I20250114 08:59:22.936872 23796 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:22.861096 23786 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:22.849972 24025 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.026731 24030 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.102228 24065 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.096230 23997 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:23.105084 21596 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.381s user 0.020s sys 0.031s
W20250114 08:59:23.152702 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 1.42933e+06us
W20250114 08:59:23.122274 24022 log.cc:926] Time spent T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.968s user 0.000s sys 0.001s
I20250114 08:59:23.104561 23913 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.066092 23783 consensus_queue.cc:1046] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20250114 08:59:23.172995 24021 log.cc:926] Time spent T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.581s user 0.001s sys 0.000s
I20250114 08:59:23.110091 24051 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.116400 23467 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.089340 23472 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.151223 24036 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:23.205513 24037 log.cc:926] Time spent T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.998s user 0.001s sys 0.000s
W20250114 08:59:23.221022 24046 log.cc:926] Time spent T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.811s user 0.001s sys 0.000s
I20250114 08:59:23.173389 23792 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:23.222417 23793 consensus_meta.cc:287] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.549s user 0.011s sys 0.004s
I20250114 08:59:23.092041 23907 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:23.258224 21596 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:23.250805 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 157179us
I20250114 08:59:23.261757 24059 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.221941 24033 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.221923 24052 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:23.267151 23913 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:23.266002 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.289485 21596 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:23.271044 23903 raft_consensus.cc:3049] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:23.272429 23783 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.306152 21629 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
W20250114 08:59:23.301270 21669 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.007s sys 0.006s
I20250114 08:59:23.281136 23784 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.313015 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
W20250114 08:59:23.314144 21669 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.007s sys 0.006s
I20250114 08:59:23.281855 24074 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:23.317184 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.317660 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.319028 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.320717 21625 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:23.323563 21596 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:23.308162 21547 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.925s user 0.000s sys 0.014s
W20250114 08:59:23.322515 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 570491us
W20250114 08:59:23.307137 24034 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.301s user 0.000s sys 0.001s
W20250114 08:59:23.350981 21547 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s user 0.000s sys 0.017s
I20250114 08:59:23.298822 23907 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:23.306416 23786 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.272701 23467 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.300879 23793 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:23.272882 24015 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.008s sys 0.007s
I20250114 08:59:23.325058 23783 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.384032 23783 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.388839 21624 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
W20250114 08:59:23.386148 24046 log.cc:926] Time spent T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20250114 08:59:23.290943 23792 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.387816 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1328) took 1833 ms. Trace:
I20250114 08:59:23.415436 21626 rpcz_store.cc:276] 0114 08:59:21.554642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.555345 (+ 703us) service_pool.cc:225] Handling call
0114 08:59:21.555816 (+ 471us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:21.556223 (+ 407us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.556264 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:21.556995 (+ 731us) log.cc:844] Serialized 52 byte log entry
0114 08:59:21.851842 (+294847us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:22.119630 (+267788us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.146604 (+ 26974us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.170289 (+ 23685us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:22.171113 (+ 824us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:23.387528 (+1216415us) raft_consensus.cc:1680] finished
0114 08:59:23.387555 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:23.387745 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":211425,"prepare.queue_time_us":77454,"prepare.run_cpu_time_us":5773,"prepare.run_wall_time_us":98808,"raft.queue_time_us":48024,"raft.run_cpu_time_us":450,"raft.run_wall_time_us":41006,"spinlock_wait_cycles":96249216,"thread_start_us":462506,"threads_started":2,"wal-append.queue_time_us":469126}
I20250114 08:59:23.312990 24064 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:23.425225 24045 log.cc:926] Time spent T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
W20250114 08:59:23.425829 23936 consensus_meta.cc:287] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.790s user 0.012s sys 0.003s
I20250114 08:59:23.426263 23936 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:23.426817 24048 log.cc:926] Time spent T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.253s user 0.000s sys 0.000s
I20250114 08:59:23.427575 21630 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:23.385715 23937 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:23.313599 24063 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.357414 21596 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:23.435842 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 78536.2us
I20250114 08:59:23.392879 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:23.396014 23783 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:23.395804 24084 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:23.150744 24065 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.308024 24052 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.447136 24076 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.447726 24076 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.281754 24036 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.448691 24065 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.448529 24063 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:23.424490 24060 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.152s user 0.000s sys 0.000s
I20250114 08:59:23.450027 24036 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.450795 24063 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.354233 24074 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.307719 24072 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:23.451346 24063 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.451503 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.451718 24072 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.307952 23903 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:23.452767 24074 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.453519 24074 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:23.090791 24050 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.453382 23903 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.425392 24064 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.454452 23903 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:23.450030 24076 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.455807 24064 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.354141 24057 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:23.050496 24039 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:23.438241 21593 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.010s sys 0.019s
I20250114 08:59:23.440668 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1343) took 2412 ms. Trace:
I20250114 08:59:23.472626 21696 rpcz_store.cc:276] 0114 08:59:21.016688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.304701 (+288013us) service_pool.cc:225] Handling call
0114 08:59:21.354805 (+ 50104us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.355152 (+ 347us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.355179 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.379246 (+ 24067us) log.cc:844] Serialized 26 byte log entry
0114 08:59:21.954235 (+574989us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:22.025793 (+ 71558us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.026542 (+ 749us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:22.026739 (+ 197us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:23.428597 (+1401858us) raft_consensus.cc:1680] finished
0114 08:59:23.428619 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:23.428756 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":81058,"prepare.run_cpu_time_us":31169,"prepare.run_wall_time_us":83667,"raft.queue_time_us":661,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":23337,"thread_start_us":574554,"threads_started":1,"wal-append.queue_time_us":1705831}
I20250114 08:59:23.270390 24051 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.450652 24036 consensus_queue.cc:1035] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 08:59:23.454006 24072 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.355031 21629 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:23.353587 24070 raft_consensus.cc:3049] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:23.436620 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:23.426688 23915 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.391335 24069 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:23.476426 21593 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.011s sys 0.019s
I20250114 08:59:23.451872 24063 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.398102 23950 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.305541 24033 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.304013 24059 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:23.448225 24052 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.334019 23784 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.394474 21594 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:23.390151 24015 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.096s user 0.009s sys 0.008s
I20250114 08:59:23.456025 24050 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.437755 23936 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.360152 23907 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:23.456370 24064 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.436507 24025 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.471762 23937 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:23.397760 23796 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:23.508780 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 72266.4us
W20250114 08:59:23.488196 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 241111us
W20250114 08:59:23.575501 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 286801us
I20250114 08:59:23.441579 23997 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:23.534013 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 139625us
I20250114 08:59:23.534101 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.397909 23786 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 08:59:23.537952 21670 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.000s sys 0.014s
W20250114 08:59:23.584849 21670 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.030s user 0.000s sys 0.016s
I20250114 08:59:23.586346 21596 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:23.499078 24070 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:23.500785 24030 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.454109 24074 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.518406 24057 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:23.524097 23915 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:23.606295 23915 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.606815 23915 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:23.596985 21623 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:23.598264 21629 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20250114 08:59:23.600173 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.519912 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.601857 21596 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:23.590308 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.590495 23472 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.598914 24057 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.590726 24054 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.537359 24072 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.592448 24079 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:23.602077 23786 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:23.599953 24050 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.626219 24050 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.626535 23783 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:23.626242 24054 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.628952 24072 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.630895 24054 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.614734 23796 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:23.631232 24072 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.633167 24070 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.633783 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.634598 24070 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:23.628366 24055 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:23.600494 23937 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.630831 24074 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:23.640252 21520 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.093s (SENT) [suppressed 38 similar messages]
I20250114 08:59:23.584115 23913 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.621732 24057 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:23.628280 24050 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:23.678622 21518 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.026s sys 0.033s
I20250114 08:59:23.636219 24079 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:23.680944 21518 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.027s sys 0.033s
I20250114 08:59:23.581928 24064 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.633783 24039 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.642498 24059 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:23.644526 24033 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.507261 24069 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.643050 24015 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.681993 24052 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.643384 24054 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.646242 24072 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20250114 08:59:23.643748 23786 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:23.676792 23467 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.679308 23936 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.681883 23793 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.685248 24079 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:23.701107 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.685285 23950 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:23.701942 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 309151us
I20250114 08:59:23.695160 24025 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:23.719374 24071 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.720638 24054 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.721109 24069 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.721650 24051 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.713711 24015 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.723346 24063 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.703820 24039 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.723655 23915 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.705178 24059 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.697520 24042 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:23.725250 24073 log.cc:926] Time spent T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20250114 08:59:23.676508 24055 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.038s
W20250114 08:59:23.729408 21668 connection.cc:232] RPC call timeout handler was delayed by -1.451448980999885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18330000000000002s for a response
W20250114 08:59:23.807873 21668 connection.cc:232] RPC call timeout handler was delayed by -1.4288994720000119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
W20250114 08:59:23.808426 21668 connection.cc:232] RPC call timeout handler was delayed by -1.3422408569999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20250114 08:59:23.808900 21668 connection.cc:232] RPC call timeout handler was delayed by -1.2489086439999255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20250114 08:59:23.809302 21668 connection.cc:232] RPC call timeout handler was delayed by -1.115023640000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1839s for a response
W20250114 08:59:23.809710 21668 connection.cc:232] RPC call timeout handler was delayed by -1.1018909489999942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
W20250114 08:59:23.810098 21668 connection.cc:232] RPC call timeout handler was delayed by -1.0783601910000016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20250114 08:59:23.733848 23973 raft_consensus.cc:3049] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:23.735153 24064 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.736070 24052 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.711122 23935 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.740084 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1376) took 1569 ms. Trace:
I20250114 08:59:23.739960 24074 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250114 08:59:23.740845 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 63314.8us
I20250114 08:59:23.783371 24025 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.742895 24072 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:23.744036 23781 consensus_meta.cc:287] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.318s user 0.011s sys 0.003s
I20250114 08:59:23.704674 23792 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.748718 23796 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.641300 23937 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:23.754599 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 71422.5us
I20250114 08:59:23.771836 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.814288 23903 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:23.773782 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.778321 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.778963 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.779091 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.763633 23936 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.820103 23893 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:23.813696 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 232528us
I20250114 08:59:23.860284 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.861012 21629 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:23.793450 23793 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:23.741384 24030 consensus_queue.cc:1046] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:23.861850 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:23.862643 21629 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:23.780349 23950 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.829317 21668 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:23.812345 24055 consensus_queue.cc:1225] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 7.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250114 08:59:23.834985 21521 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:23.869534 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:23.811208 24059 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.812479 23973 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:23.795228 23784 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.841140 21622 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:23.878096 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 108124us
I20250114 08:59:23.772835 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.816848 23993 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.882050 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.843822 21625 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:23.882685 21696 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 3.
I20250114 08:59:23.883301 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.883810 21628 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:23.885524 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.887439 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:23.846791 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 206554us
W20250114 08:59:23.847541 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 285961us
I20250114 08:59:23.827731 23913 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.878466 23467 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.866310 23893 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.838744 24063 consensus_queue.cc:1046] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250114 08:59:23.799830 24039 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.859102 23796 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.869061 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:23.812711 24069 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.870951 21668 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:23.820547 21547 rpcz_store.cc:276] 0114 08:59:22.170365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:22.170637 (+ 272us) service_pool.cc:225] Handling call
0114 08:59:22.171045 (+ 408us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:22.171339 (+ 294us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:22.363126 (+191787us) log.cc:844] Serialized 14 byte log entry
0114 08:59:23.639496 (+1276370us) log.cc:844] Serialized 14 byte log entry
0114 08:59:23.739576 (+100080us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b1800122aa0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c60c03a 00007fa645ac8be9 00007fa63c4cd930 00007fa63c691a14 00007fa63c691305 00007fa63f5c53d0 00007fa63f5d7666 00007fa63f5e81d7 00007fa63f5e8158
0114 08:59:23.739761 (+ 185us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:23.739808 (+ 47us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:23.739853 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:23.739916 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:23.740029 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61806,"prepare.queue_time_us":194571,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":118,"raft.queue_time_us":201996,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":159,"spinlock_wait_cycles":4075392,"thread_start_us":1271239,"threads_started":2,"wal-append.queue_time_us":1218015}
I20250114 08:59:23.878204 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.876112 24051 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.785203 24071 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.936237 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.897711 21629 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:23.899726 21697 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20250114 08:59:23.900502 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 65615.3us
I20250114 08:59:23.842265 24096 consensus_queue.cc:1035] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20250114 08:59:23.871240 24081 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:23.904031 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.903991 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:23.907765 24058 log.cc:926] Time spent T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20250114 08:59:23.908064 24015 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.908567 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.912639 24093 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.912993 24091 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:23.915171 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.957160 21701 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:23.916184 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.959213 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:23.959939 21695 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:23.835738 24072 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.963528 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.917245 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.877642 24050 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:23.923458 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:23.877838 24095 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:23.840955 23781 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.870599 23793 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.825558 24087 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20250114 08:59:23.941543 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:23.974380 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:23.929456 23702 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:23.822454 24042 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.979539 21669 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:23.945495 21696 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:23.945796 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:23.916771 23899 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:23.998878 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:23.997256 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.000639 21554 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:23.946962 24051 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.946259 21625 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:23.952492 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:23.954214 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 83321.2us
I20250114 08:59:23.955132 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.864778 23903 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.964407 21697 raft_consensus.cc:2405] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 9 because replica has last-logged OpId of term: 6 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:23.889812 23993 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.938069 24074 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:23.974998 21701 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:23.905593 23973 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.939397 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.982003 23907 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.982901 24033 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:23.983582 23913 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 08:59:23.984620 24099 log.cc:926] Time spent T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250114 08:59:23.985255 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1331) took 2410 ms. Trace:
I20250114 08:59:23.987139 23796 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.989104 23793 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.968070 24091 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.989533 24059 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:23.990671 23781 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:23.963863 24093 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.921437 23467 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:23.995908 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:23.996367 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.996523 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:23.996981 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:23.997752 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:23.956432 23915 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:23.956331 23894 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:23.998590 21702 raft_consensus.cc:2388] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:23.947156 24069 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:23.940294 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.993521 24057 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:24.347064 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1711) took 2910 ms. Trace:
I20250114 08:59:24.320309 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.001120 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:24.141119 21630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1329) took 2579 ms (client timeout 2225 ms). Trace:
I20250114 08:59:24.293776 21520 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:24.358095 21630 rpcz_store.cc:269] 0114 08:59:21.561573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.561692 (+ 119us) service_pool.cc:225] Handling call
0114 08:59:24.140898 (+2579206us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b540039dd54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:24.141046 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4288427392}
W20250114 08:59:24.358810 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 65128.7us
I20250114 08:59:24.017688 21703 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:24.273133 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1713) took 2833 ms. Trace:
I20250114 08:59:24.065375 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:24.002933 21695 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
W20250114 08:59:24.292649 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 314877us
I20250114 08:59:24.369185 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.369405 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:24.357350 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 343347us
I20250114 08:59:24.361315 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:24.370209 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:24.370661 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:24.370865 21623 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:24.368261 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.366245 21550 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:24.001605 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.372423 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.373657 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.373848 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.367465 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:24.369822 21521 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:24.356181 21700 rpcz_store.cc:276] 0114 08:59:21.437051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.437190 (+ 139us) service_pool.cc:225] Handling call
0114 08:59:21.445591 (+ 8401us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.458930 (+ 13339us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.458961 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.459417 (+ 456us) log.cc:844] Serialized 26 byte log entry
0114 08:59:21.738085 (+278668us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:21.738126 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:21.738204 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:24.345841 (+2607637us) spinlock_profiling.cc:243] Waited 64.1 ms on lock 0x7b2400245b28. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c4eb9d4 00007fa63c4e9adb 00007fa63c578289 00007fa63c577f15 00007fa63c577470 00007fa63f277f88 00007fa63f2786e8 00007fa63f24ff98 00007fa63f24b3aa 00007fa63f24a7f6
0114 08:59:24.346595 (+ 754us) raft_consensus.cc:1680] finished
0114 08:59:24.346630 (+ 35us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:24.346974 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13482,"raft.queue_time_us":23575,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":13256,"spinlock_wait_cycles":147420544,"thread_start_us":230793,"threads_started":1,"wal-append.queue_time_us":397811}
I20250114 08:59:24.366869 21627 rpcz_store.cc:276] 0114 08:59:21.574525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.576319 (+ 1794us) service_pool.cc:225] Handling call
0114 08:59:21.576748 (+ 429us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.988195 (+411447us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.988226 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.989202 (+ 976us) log.cc:844] Serialized 26 byte log entry
0114 08:59:22.754678 (+765476us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:22.814183 (+ 59505us) log.cc:844] Serialized 14 byte log entry
0114 08:59:23.313731 (+499548us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:23.748237 (+434506us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b540027f454. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:23.748285 (+ 48us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:23.984931 (+236646us) raft_consensus.cc:1680] finished
0114 08:59:23.985033 (+ 102us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:23.985168 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56375,"prepare.queue_time_us":216507,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":569061,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":12575,"spinlock_wait_cycles":88438144,"thread_start_us":1576072,"threads_started":3,"wal-append.queue_time_us":1650225}
I20250114 08:59:24.380622 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:24.371470 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:24.109247 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:24.371788 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:23.981375 24054 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:24.462105 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:24.462379 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 85144.4us
I20250114 08:59:24.462358 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.465090 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:24.391119 21695 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:24.407733 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:24.384439 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:24.470916 21628 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:24.385130 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.466367 21623 raft_consensus.cc:2371] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
W20250114 08:59:24.062947 24014 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 2.310s user 0.004s sys 0.013s
I20250114 08:59:24.389756 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.388737 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.384909 21696 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:24.389151 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:24.379344 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:24.170956 24007 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 2.398s user 0.000s sys 0.016s
I20250114 08:59:23.999073 23899 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:24.048249 24052 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:24.347452 24042 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:24.094205 24072 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:24.069252 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:23.973471 24050 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:24.293999 23874 consensus_queue.cc:1035] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.267s
I20250114 08:59:23.998652 23702 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:25.723079 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.476881 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.734651 21549 raft_consensus.cc:2388] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:25.735395 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1512) took 1590 ms. Trace:
I20250114 08:59:25.732741 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1511) took 1685 ms. Trace:
I20250114 08:59:25.736852 21548 rpcz_store.cc:276] 0114 08:59:24.046925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.388368 (+341443us) service_pool.cc:225] Handling call
0114 08:59:25.732393 (+1344025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.735821 21549 rpcz_store.cc:276] 0114 08:59:24.144765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.476599 (+331834us) service_pool.cc:225] Handling call
0114 08:59:25.735271 (+1258672us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:24.503258 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 95688.7us
I20250114 08:59:24.477625 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.759128 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1387) took 1468 ms. Trace:
I20250114 08:59:25.792647 21553 rpcz_store.cc:276] 0114 08:59:24.290994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.389441 (+ 98447us) service_pool.cc:225] Handling call
0114 08:59:25.758829 (+1369388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.793218 21554 raft_consensus.cc:2388] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:24.680246 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 305395us
I20250114 08:59:25.793798 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1513) took 1331 ms. Trace:
I20250114 08:59:25.794217 21554 rpcz_store.cc:276] 0114 08:59:24.463300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.477401 (+ 14101us) service_pool.cc:225] Handling call
0114 08:59:25.793715 (+1316314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.767894 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.773952 21622 raft_consensus.cc:2405] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20250114 08:59:25.779767 21624 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s user 0.005s sys 0.015s
I20250114 08:59:24.561228 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:25.531273 21669 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.003s sys 0.006s
I20250114 08:59:25.797344 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1515) took 1232 ms. Trace:
W20250114 08:59:25.810109 21594 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.009s sys 0.003s
W20250114 08:59:25.811070 21594 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.877s user 0.009s sys 0.004s
W20250114 08:59:25.811820 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 877652us
I20250114 08:59:25.808641 21555 rpcz_store.cc:276] 0114 08:59:24.564668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.767057 (+1202389us) service_pool.cc:225] Handling call
0114 08:59:25.797240 (+ 30183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:24.357376 23824 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:25.847179 21624 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.347s user 0.006s sys 0.016s
I20250114 08:59:24.366252 21704 rpcz_store.cc:276] 0114 08:59:21.439995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.440188 (+ 193us) service_pool.cc:225] Handling call
0114 08:59:21.440610 (+ 422us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.441122 (+ 512us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.441162 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.460684 (+ 19522us) log.cc:844] Serialized 26 byte log entry
0114 08:59:21.738677 (+277993us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:22.838964 (+1100287us) log.cc:844] Serialized 14 byte log entry
0114 08:59:24.265170 (+1426206us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b180010a6e0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63f25808c 00007fa63c676bfc 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c691305 00007fa63f5c53d0 00007fa63f5d7666 00007fa63f5e81d7
0114 08:59:24.265328 (+ 158us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:24.265425 (+ 97us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:24.265485 (+ 60us) raft_consensus.cc:1680] finished
0114 08:59:24.272856 (+ 7371us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:24.273053 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57783,"prepare.queue_time_us":161118,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":6956,"raft.queue_time_us":271359,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":10708,"spinlock_wait_cycles":8443136,"thread_start_us":1647380,"threads_started":3,"wal-append.queue_time_us":490928,"wal-append.run_cpu_time_us":13076,"wal-append.run_wall_time_us":2116992}
I20250114 08:59:24.365761 23893 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:25.870114 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.870842 23893 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:25.871269 21704 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:25.875624 24050 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:25.869652 22269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 379) took 1867 ms (client timeout 1776 ms). Trace:
I20250114 08:59:24.068678 24094 raft_consensus.cc:3049] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
W20250114 08:59:25.876710 22269 rpcz_store.cc:269] 0114 08:59:24.002272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.013267 (+ 10995us) service_pool.cc:225] Handling call
0114 08:59:25.869257 (+1855990us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.876955 24094 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:24.188861 24096 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:25.878762 21669 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.468s user 0.004s sys 0.007s
W20250114 08:59:25.879554 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 1.49886e+06us
I20250114 08:59:25.880316 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.189575 23936 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.881996 23936 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:25.882630 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.882990 23936 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:25.884570 24011 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:25.885174 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.885594 24011 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:24.003367 24039 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:24.366564 23431 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:24.383087 23915 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:25.885273 21696 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1364) took 2000 ms (client timeout 1508 ms). Trace:
I20250114 08:59:24.463019 24000 consensus_queue.cc:1046] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 1.508s
I20250114 08:59:24.308760 23472 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:24.033661 24081 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 1.858s
I20250114 08:59:25.884419 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:24.142855 23950 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.888427 23431 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:24.366215 23937 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:25.889107 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.897626 21553 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
W20250114 08:59:24.765810 24105 log.cc:926] Time spent T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.452s user 0.000s sys 0.000s
I20250114 08:59:25.900523 21628 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:25.901405 21627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1567) took 1517 ms (client timeout 1972 ms). Trace:
I20250114 08:59:25.893250 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 141) took 4883 ms. Trace:
W20250114 08:59:25.903450 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1374) took 1984 ms (client timeout 1931 ms). Trace:
W20250114 08:59:25.904189 21701 rpcz_store.cc:269] 0114 08:59:23.919015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.356115 (+437100us) service_pool.cc:225] Handling call
0114 08:59:25.903191 (+1547076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.894686 23899 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:25.905931 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:25.906867 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1380) took 1985 ms (client timeout 2234 ms). Trace:
W20250114 08:59:25.907251 21701 rpcz_store.cc:269] 0114 08:59:23.921294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.905431 (+1984137us) service_pool.cc:225] Handling call
0114 08:59:25.906796 (+ 1365us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:25.894840 21622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1565) took 1982 ms (client timeout 2634 ms). Trace:
I20250114 08:59:24.549469 23781 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 1.360s
W20250114 08:59:25.909209 21622 rpcz_store.cc:269] 0114 08:59:23.913117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.379098 (+465981us) service_pool.cc:225] Handling call
0114 08:59:25.861568 (+1482470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.255910 23993 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:24.557070 24051 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:24.562654 23984 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:25.902300 21699 rpcz_store.cc:276] 0114 08:59:21.009247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:21.009527 (+ 280us) service_pool.cc:225] Handling call
0114 08:59:21.029133 (+ 19606us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:21.035238 (+ 6105us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:21.035266 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:21.049778 (+ 14512us) log.cc:844] Serialized 26 byte log entry
0114 08:59:21.436775 (+386997us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:21.663932 (+227157us) log.cc:844] Serialized 14 byte log entry
0114 08:59:22.044154 (+380222us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:22.044256 (+ 102us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:24.092946 (+2048690us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:25.892451 (+1799505us) raft_consensus.cc:1680] finished
0114 08:59:25.892649 (+ 198us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:25.892955 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5667,"prepare.queue_time_us":368581,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":56580,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":49024,"spinlock_wait_cycles":93737344,"thread_start_us":884503,"threads_started":3,"wal-append.queue_time_us":510222}
I20250114 08:59:25.400175 23934 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:25.914443 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.914253 23934 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:25.915269 21699 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:25.915781 21699 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1743) took 1445 ms (client timeout 1852 ms). Trace:
I20250114 08:59:25.236398 24069 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:25.238298 23467 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:24.503822 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.209088 23907 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:25.918722 21697 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1373) took 2000 ms (client timeout 2071 ms). Trace:
I20250114 08:59:24.545995 23894 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 1.373s
W20250114 08:59:25.919203 21697 rpcz_store.cc:269] 0114 08:59:23.918359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.373896 (+455537us) service_pool.cc:225] Handling call
0114 08:59:25.918494 (+1544598us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:25.528815 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 356134us
W20250114 08:59:25.890439 21696 rpcz_store.cc:269] 0114 08:59:23.884718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.885060 (+ 342us) service_pool.cc:225] Handling call
0114 08:59:25.884959 (+1999899us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:24.368299 23913 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:25.916131 21699 rpcz_store.cc:269] 0114 08:59:24.469967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.913948 (+1443981us) service_pool.cc:225] Handling call
0114 08:59:25.915724 (+ 1776us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:24.379381 24074 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:25.912905 23984 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:25.882886 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 86611.2us
I20250114 08:59:25.912349 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1528) took 2085 ms. Trace:
I20250114 08:59:25.903506 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1341) took 2309 ms. Trace:
I20250114 08:59:25.879477 24096 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:25.872162 21704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1741) took 1403 ms (client timeout 1533 ms). Trace:
W20250114 08:59:25.901948 21627 rpcz_store.cc:269] 0114 08:59:24.383476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.432778 (+ 49302us) service_pool.cc:225] Handling call
0114 08:59:25.901120 (+1468342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.929137 21622 rpcz_store.cc:276] 0114 08:59:23.826434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.910693 (+2084259us) service_pool.cc:225] Handling call
0114 08:59:25.911378 (+ 685us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:25.911864 (+ 486us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:25.911897 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:25.911963 (+ 66us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:25.912034 (+ 71us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:25.912115 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:25.912248 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.908277 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.932901 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:25.930663 21704 rpcz_store.cc:269] 0114 08:59:24.468868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.869310 (+1400442us) service_pool.cc:225] Handling call
0114 08:59:25.872047 (+ 2737us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:24.362612 23972 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250114 08:59:25.250244 24112 log.cc:926] Time spent T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250114 08:59:25.109324 24113 log.cc:926] Time spent T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.372s user 0.000s sys 0.000s
I20250114 08:59:25.936622 23984 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:25.937047 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:25.900100 23431 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.937399 24000 consensus_queue.cc:1225] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20250114 08:59:25.243391 24079 consensus_meta.cc:287] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.466s user 0.010s sys 0.002s
W20250114 08:59:24.731581 23786 consensus_meta.cc:287] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.973s user 0.010s sys 0.002s
I20250114 08:59:25.940641 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:25.900104 23702 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:24.378751 24095 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:24.619223 24033 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:25.895260 24011 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.900384 23935 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:25.871820 23893 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:24.559825 24057 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.927031 24050 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:25.954829 23936 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:24.384419 24075 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:25.953995 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:24.614337 24054 consensus_queue.cc:1046] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 1.334s
I20250114 08:59:24.558454 24091 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:24.619246 24116 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:24.560865 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.894147 24081 consensus_queue.cc:1035] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.069s
I20250114 08:59:25.973546 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.972308 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.235826 24093 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.237298 23903 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:25.983330 23786 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:25.985834 23786 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:25.987179 23899 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:25.988586 21546 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 10.
I20250114 08:59:25.989703 21520 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:25.981215 24079 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:25.982401 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:25.987591 24096 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:25.980602 21697 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:25.929574 21626 rpcz_store.cc:276] 0114 08:59:23.593688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.594252 (+ 564us) service_pool.cc:225] Handling call
0114 08:59:23.594719 (+ 467us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:23.595229 (+ 510us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:23.595268 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:23.595972 (+ 704us) log.cc:844] Serialized 52 byte log entry
0114 08:59:23.818076 (+222104us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:23.818132 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:23.818227 (+ 95us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:25.902933 (+2084706us) raft_consensus.cc:1680] finished
0114 08:59:25.902970 (+ 37us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:25.903275 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5508,"raft.queue_time_us":3937,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":174089,"threads_started":1,"wal-append.queue_time_us":525829}
I20250114 08:59:25.992267 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:26.002949 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 71525.7us
I20250114 08:59:25.998930 21701 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:25.999728 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1525) took 2174 ms. Trace:
I20250114 08:59:26.000696 23934 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:26.000715 21697 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1371) took 2083 ms (client timeout 2709 ms). Trace:
W20250114 08:59:26.005224 21697 rpcz_store.cc:269] 0114 08:59:23.917767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.920070 (+2002303us) service_pool.cc:225] Handling call
0114 08:59:26.000637 (+ 80567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:25.997383 24000 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:25.966827 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:25.979177 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:25.892117 23472 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:25.889520 23915 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.010967 21704 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:26.014127 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1382) took 2069 ms. Trace:
I20250114 08:59:26.014521 21704 rpcz_store.cc:276] 0114 08:59:23.944550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.942304 (+1997754us) service_pool.cc:225] Handling call
0114 08:59:26.014051 (+ 71747us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.001931 21626 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:25.999127 23893 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.017465 21621 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:26.018399 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1749) took 1544 ms. Trace:
I20250114 08:59:26.018923 21696 rpcz_store.cc:276] 0114 08:59:24.473961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.945955 (+1471994us) service_pool.cc:225] Handling call
0114 08:59:26.018316 (+ 72361us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1057536}
I20250114 08:59:26.009450 24081 consensus_queue.cc:1225] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:26.022619 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1531) took 2192 ms. Trace:
I20250114 08:59:26.007555 24079 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.024581 21699 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:26.024968 21626 rpcz_store.cc:276] 0114 08:59:23.827646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.000915 (+2173269us) service_pool.cc:225] Handling call
0114 08:59:26.001415 (+ 500us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.015540 (+ 14125us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.019980 (+ 4440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.031352 24054 raft_consensus.cc:3049] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:26.031860 24054 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:26.032974 21668 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.033211 24054 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.035158 24054 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:26.035917 21668 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:26.041034 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.026228 22269 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.004093 21624 rpcz_store.cc:276] 0114 08:59:23.825454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.470865 (+645411us) service_pool.cc:225] Handling call
0114 08:59:24.471179 (+ 314us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:24.472129 (+ 950us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:24.500099 (+ 27970us) log.cc:844] Serialized 14 byte log entry
0114 08:59:25.958379 (+1458280us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:25.969744 (+ 11365us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:25.999326 (+ 29582us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:25.999426 (+ 100us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:25.999638 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27071,"prepare.queue_time_us":584922,"prepare.run_cpu_time_us":10749,"prepare.run_wall_time_us":758988,"raft.queue_time_us":6047,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":49510,"thread_start_us":1420406,"threads_started":1}
I20250114 08:59:26.023543 24117 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.036805 24033 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.043502 21626 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:59:26.051280 24000 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.012744 23913 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:26.028537 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1746) took 1555 ms. Trace:
I20250114 08:59:26.061877 21699 rpcz_store.cc:276] 0114 08:59:24.473024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.950377 (+1477353us) service_pool.cc:225] Handling call
0114 08:59:26.028439 (+ 78062us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.060566 21697 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:26.063356 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1377) took 2143 ms. Trace:
I20250114 08:59:26.062987 21700 raft_consensus.cc:2388] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:26.063756 21697 rpcz_store.cc:276] 0114 08:59:23.920008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.012901 (+2092893us) service_pool.cc:225] Handling call
0114 08:59:26.063266 (+ 50365us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.065691 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1533) took 2235 ms. Trace:
I20250114 08:59:26.066212 21626 rpcz_store.cc:276] 0114 08:59:23.830450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.042187 (+2211737us) service_pool.cc:225] Handling call
0114 08:59:26.042845 (+ 658us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.061577 (+ 18732us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.065617 (+ 4040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.021049 24050 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:26.074960 21700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1375) took 2155 ms (client timeout 2223 ms). Trace:
I20250114 08:59:26.076119 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1529) took 2247 ms. Trace:
I20250114 08:59:26.077901 21627 rpcz_store.cc:276] 0114 08:59:23.826757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.951789 (+2125032us) service_pool.cc:225] Handling call
0114 08:59:26.017919 (+ 66130us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.073736 (+ 55817us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.073777 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.073843 (+ 66us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:26.073903 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.073982 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.074113 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:26.076643 21700 rpcz_store.cc:269] 0114 08:59:23.919388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.432534 (+513146us) service_pool.cc:225] Handling call
0114 08:59:26.072401 (+1639867us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.093393 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.099222 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1535) took 2266 ms. Trace:
I20250114 08:59:26.081935 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1383) took 2114 ms. Trace:
I20250114 08:59:26.133257 21624 rpcz_store.cc:276] 0114 08:59:23.833037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.080254 (+2247217us) service_pool.cc:225] Handling call
0114 08:59:26.098293 (+ 18039us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.098652 (+ 359us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.098693 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.098774 (+ 81us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.098869 (+ 95us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.098942 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.099124 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.115837 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1362) took 2235 ms. Trace:
I20250114 08:59:26.108573 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.080199 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:26.127836 21703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1368) took 2210 ms (client timeout 2722 ms). Trace:
W20250114 08:59:26.115314 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 160137us
I20250114 08:59:26.133953 21700 rpcz_store.cc:276] 0114 08:59:23.967587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.080767 (+2113180us) service_pool.cc:225] Handling call
0114 08:59:26.081209 (+ 442us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.081519 (+ 310us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.081544 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.081617 (+ 73us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.081667 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.081713 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.081842 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.103173 21521 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.149312 21704 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250114 08:59:26.141232 21703 rpcz_store.cc:269] 0114 08:59:23.916881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.917003 (+ 122us) service_pool.cc:225] Handling call
0114 08:59:26.127348 (+2210345us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b54001c0a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.127609 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121174400}
I20250114 08:59:26.105005 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:26.080029 23899 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.116704 24033 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:26.125137 24075 raft_consensus.cc:695] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.134305 21698 rpcz_store.cc:276] 0114 08:59:23.880767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.881038 (+ 271us) service_pool.cc:225] Handling call
0114 08:59:23.883705 (+ 2667us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:23.884078 (+ 373us) raft_consensus.cc:1529] Early marking committed up to index 1
0114 08:59:23.884103 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:23.884678 (+ 575us) log.cc:844] Serialized 52 byte log entry
0114 08:59:23.968884 (+ 84206us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:23.968910 (+ 26us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:23.968966 (+ 56us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:26.069606 (+2100640us) raft_consensus.cc:1680] finished
0114 08:59:26.072662 (+ 3056us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.115585 (+ 42923us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1859,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":9097,"thread_start_us":83943,"threads_started":1,"wal-append.queue_time_us":942612}
I20250114 08:59:26.118052 23903 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 08:59:26.155993 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 120296us
I20250114 08:59:26.106989 22269 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:26.157671 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1378) took 2234 ms. Trace:
I20250114 08:59:26.092056 23702 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:26.161845 24075 consensus_queue.cc:237] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.092675 24116 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:26.157271 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1742) took 1687 ms. Trace:
I20250114 08:59:26.168787 21696 rpcz_store.cc:276] 0114 08:59:24.469407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.026608 (+1557201us) service_pool.cc:225] Handling call
0114 08:59:26.157220 (+130612us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.159154 23915 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.165273 21704 rpcz_store.cc:276] 0114 08:59:23.920659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.030357 (+2109698us) service_pool.cc:225] Handling call
0114 08:59:26.154630 (+124273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.170702 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:26.168141 22274 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:26.153899 24079 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.160351 23472 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.180593 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.181828 23899 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:26.182039 24053 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:26.182377 23899 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:26.182680 24050 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.182862 21624 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:26.183665 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1537) took 2350 ms. Trace:
I20250114 08:59:26.183740 21698 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:26.183534 24000 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.184108 21624 rpcz_store.cc:276] 0114 08:59:23.835776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.140538 (+2304762us) service_pool.cc:225] Handling call
0114 08:59:26.179277 (+ 38739us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.183338 (+ 4061us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.183586 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.186079 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1538) took 2346 ms. Trace:
I20250114 08:59:26.186420 21624 rpcz_store.cc:276] 0114 08:59:23.840052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.185001 (+2344949us) service_pool.cc:225] Handling call
0114 08:59:26.185395 (+ 394us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.185745 (+ 350us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.185770 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.185811 (+ 41us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:26.185858 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.185917 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.186018 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.187417 24074 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:26.188207 24070 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:26.182893 23899 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.189024 24070 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.189857 23912 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:26.190420 23912 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.191506 24070 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:26.191707 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1737) took 1725 ms. Trace:
I20250114 08:59:26.192247 24127 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:26.193212 24127 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.192140 21703 rpcz_store.cc:276] 0114 08:59:24.466553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.170835 (+1704282us) service_pool.cc:225] Handling call
0114 08:59:26.187966 (+ 17131us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.188238 (+ 272us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.188261 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.188297 (+ 36us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:26.188336 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.188387 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.191625 (+ 3238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.188311 21624 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:26.196789 21626 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
W20250114 08:59:26.201558 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 109318us
W20250114 08:59:26.202162 21621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1552) took 2293 ms (client timeout 1522 ms). Trace:
I20250114 08:59:26.202500 21518 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:26.202615 21621 rpcz_store.cc:269] 0114 08:59:23.908932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.935902 (+ 26970us) service_pool.cc:225] Handling call
0114 08:59:26.201872 (+2265970us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54002de1d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.202044 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4750468480}
I20250114 08:59:26.182145 21554 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:26.198195 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1738) took 1717 ms. Trace:
I20250114 08:59:26.205559 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:26.210911 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 45853.5us
I20250114 08:59:26.211220 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1539) took 2358 ms. Trace:
I20250114 08:59:26.211985 21624 rpcz_store.cc:276] 0114 08:59:23.846968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.187448 (+2340480us) service_pool.cc:225] Handling call
0114 08:59:26.187894 (+ 446us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.198111 (+ 10217us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.198379 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.209350 21698 rpcz_store.cc:276] 0114 08:59:24.467198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.174508 (+1707310us) service_pool.cc:225] Handling call
0114 08:59:26.183287 (+ 8779us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.184236 (+ 949us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.184425 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:26.208305 21623 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.102s user 0.062s sys 0.084s
I20250114 08:59:26.217592 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1534) took 2376 ms. Trace:
I20250114 08:59:26.218127 21626 rpcz_store.cc:276] 0114 08:59:23.832702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.071848 (+2239146us) service_pool.cc:225] Handling call
0114 08:59:26.165485 (+ 93637us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.208678 (+ 43193us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.208843 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.216500 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1540) took 2361 ms. Trace:
I20250114 08:59:26.178148 23984 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.269743 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1384) took 2291 ms. Trace:
W20250114 08:59:26.237179 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 50378.4us
I20250114 08:59:26.221202 21621 rpcz_store.cc:276] 0114 08:59:23.847286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.203502 (+2356216us) service_pool.cc:225] Handling call
0114 08:59:26.203963 (+ 461us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.204314 (+ 351us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.204345 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.204392 (+ 47us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.204428 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.208729 (+ 4301us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.208830 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.298316 21700 rpcz_store.cc:276] 0114 08:59:23.978683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.165563 (+2186880us) service_pool.cc:225] Handling call
0114 08:59:26.208262 (+ 42699us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.255391 (+ 47129us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.255435 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.269253 (+ 13818us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.269349 (+ 96us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.269434 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.269625 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.273257 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.204440 21518 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.265746 21624 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 08:59:26.302932 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1541) took 2455 ms. Trace:
I20250114 08:59:26.303267 21624 rpcz_store.cc:276] 0114 08:59:23.847515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.247782 (+2400267us) service_pool.cc:225] Handling call
0114 08:59:26.265223 (+ 17441us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.302665 (+ 37442us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.302877 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:26.267702 21702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1361) took 2387 ms (client timeout 1561 ms). Trace:
I20250114 08:59:26.239200 21668 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:26.239200 21669 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:26.320611 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.321935 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.323076 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.323802 21554 raft_consensus.cc:2388] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.323558 23899 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.324961 23912 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.325531 21704 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 08:59:26.326231 24071 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.326244 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1744) took 1855 ms. Trace:
I20250114 08:59:26.327836 21704 rpcz_store.cc:276] 0114 08:59:24.470497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.174178 (+1703681us) service_pool.cc:225] Handling call
0114 08:59:26.325071 (+150893us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.325943 (+ 872us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.326154 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:26.327304 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 48047.3us
I20250114 08:59:26.179654 24033 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
W20250114 08:59:26.261981 24132 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250114 08:59:26.196440 24127 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:26.313812 21702 rpcz_store.cc:269] 0114 08:59:23.880265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.880411 (+ 146us) service_pool.cc:225] Handling call
0114 08:59:26.267229 (+2386818us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400329454. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.267502 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":253516672}
I20250114 08:59:26.334628 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1544) took 2486 ms. Trace:
W20250114 08:59:26.337486 24114 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 2.164s user 0.000s sys 0.017s
I20250114 08:59:26.340402 21624 rpcz_store.cc:276] 0114 08:59:23.848192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.317859 (+2469667us) service_pool.cc:225] Handling call
0114 08:59:26.333809 (+ 15950us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.334188 (+ 379us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.334218 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.334288 (+ 70us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.334333 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.334397 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.334544 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.345623 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1542) took 2497 ms. Trace:
I20250114 08:59:26.290757 24095 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:26.248378 24074 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.348453 21626 rpcz_store.cc:276] 0114 08:59:23.847753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.237131 (+2389378us) service_pool.cc:225] Handling call
0114 08:59:26.313855 (+ 76724us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.330998 (+ 17143us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.331040 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.331106 (+ 66us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:26.345277 (+ 14171us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.345377 (+ 100us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.345546 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.269063 24116 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:26.354024 21549 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.326819 24070 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:26.350525 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 116775us
W20250114 08:59:26.357656 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 254815us
I20250114 08:59:26.359130 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1527) took 2533 ms. Trace:
I20250114 08:59:26.359596 21623 rpcz_store.cc:276] 0114 08:59:23.826132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.472953 (+646821us) service_pool.cc:225] Handling call
0114 08:59:24.473295 (+ 342us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:24.473535 (+ 240us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:26.215546 (+1742011us) log.cc:844] Serialized 14 byte log entry
0114 08:59:26.355610 (+140064us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.358687 (+ 3077us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.358774 (+ 87us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.358853 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.359053 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58673,"prepare.queue_time_us":1438391,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":23474,"thread_start_us":1258408,"threads_started":2,"wal-append.queue_time_us":69662}
I20250114 08:59:26.362358 23912 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:26.362561 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1546) took 2505 ms. Trace:
I20250114 08:59:26.362910 23912 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:26.363420 23912 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:26.363940 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 152058us
I20250114 08:59:26.364657 23903 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:26.365636 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 161368us
I20250114 08:59:26.365906 24128 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:26.366580 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1514) took 1819 ms. Trace:
I20250114 08:59:26.355818 24116 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.367177 24116 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:26.367786 21518 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:26.363071 21623 rpcz_store.cc:276] 0114 08:59:23.856991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.361360 (+2504369us) service_pool.cc:225] Handling call
0114 08:59:26.361803 (+ 443us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.362205 (+ 402us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.362231 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.362278 (+ 47us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.362325 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.362374 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.362503 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.370458 21669 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:26.356833 24095 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:26.259958 24079 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.377537 24079 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.350991 24094 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.379464 24094 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:26.366988 21546 rpcz_store.cc:276] 0114 08:59:24.546768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.551228 (+ 4460us) service_pool.cc:225] Handling call
0114 08:59:26.366365 (+1815137us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b540016ec54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.366515 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":499900160}
I20250114 08:59:26.368419 21700 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 08:59:26.380852 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1748) took 1907 ms. Trace:
I20250114 08:59:26.381172 21700 rpcz_store.cc:276] 0114 08:59:24.473670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.304220 (+1830550us) service_pool.cc:225] Handling call
0114 08:59:26.332405 (+ 28185us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.380555 (+ 48150us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.380774 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.382714 21700 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:26.383265 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1752) took 1908 ms. Trace:
I20250114 08:59:26.383553 21700 rpcz_store.cc:276] 0114 08:59:24.474837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.381957 (+1907120us) service_pool.cc:225] Handling call
0114 08:59:26.382340 (+ 383us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.383034 (+ 694us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.383208 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.234143 23702 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.386163 23702 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.377580 23984 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:26.375692 24033 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:26.388868 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1526) took 2558 ms. Trace:
I20250114 08:59:26.397516 21621 raft_consensus.cc:1270] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 3. (term mismatch)
I20250114 08:59:26.394820 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.397949 21621 pending_rounds.cc:77] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 2
I20250114 08:59:26.395797 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.399873 21621 pending_rounds.cc:99] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250114 08:59:26.182729 24053 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.394574 21628 rpcz_store.cc:276] 0114 08:59:23.825820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.471592 (+645772us) service_pool.cc:225] Handling call
0114 08:59:24.471866 (+ 274us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:25.901402 (+1429536us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.384115 (+482713us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54002fab54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:26.384404 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3284620032}
I20250114 08:59:26.403685 21621 raft_consensus.cc:2883] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:26.407470 23973 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:26.408713 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1501) took 2452 ms. Trace:
I20250114 08:59:26.409323 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:26.409788 21695 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1369) took 2492 ms (client timeout 1945 ms). Trace:
I20250114 08:59:26.413384 21555 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:26.414294 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1740) took 1946 ms (client timeout 1797 ms). Trace:
W20250114 08:59:26.424248 21701 rpcz_store.cc:269] 0114 08:59:24.468298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.907894 (+1439596us) service_pool.cc:225] Handling call
0114 08:59:26.414080 (+506186us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b5400370fd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.414192 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61683968}
W20250114 08:59:26.419082 21695 rpcz_store.cc:269] 0114 08:59:23.917178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.369177 (+451999us) service_pool.cc:225] Handling call
0114 08:59:26.409431 (+2040254us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b5400003a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.409584 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48196352}
I20250114 08:59:26.418634 21552 rpcz_store.cc:276] 0114 08:59:23.956419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.956705 (+ 286us) service_pool.cc:225] Handling call
0114 08:59:23.957053 (+ 348us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:24.463046 (+505993us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:24.463081 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:24.463124 (+ 43us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:24.463153 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.408253 (+1945100us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400194e54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:26.408319 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.408527 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1163193088}
W20250114 08:59:26.427071 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 59377us
I20250114 08:59:26.435070 21702 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:26.434341 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1543) took 2586 ms. Trace:
I20250114 08:59:26.437371 21621 rpcz_store.cc:276] 0114 08:59:23.847983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.331106 (+2483123us) service_pool.cc:225] Handling call
0114 08:59:26.388235 (+ 57129us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.434080 (+ 45845us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.434272 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.413410 23899 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.436892 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.444372 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1751) took 1969 ms. Trace:
I20250114 08:59:26.445739 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.446657 23903 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:26.447737 24054 raft_consensus.cc:3049] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:26.448707 21668 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.449820 23903 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:26.444876 21702 rpcz_store.cc:276] 0114 08:59:24.474534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.370646 (+1896112us) service_pool.cc:225] Handling call
0114 08:59:26.426303 (+ 55657us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.440010 (+ 13707us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.444303 (+ 4293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.448012 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1553) took 2538 ms. Trace:
I20250114 08:59:26.451876 21626 rpcz_store.cc:276] 0114 08:59:23.909192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.446925 (+2537733us) service_pool.cc:225] Handling call
0114 08:59:26.447351 (+ 426us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.447625 (+ 274us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.447656 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.447730 (+ 74us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.447802 (+ 72us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.447860 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.447957 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.457424 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1376) took 2532 ms. Trace:
I20250114 08:59:26.402820 24070 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.457247 23472 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.457993 21699 rpcz_store.cc:276] 0114 08:59:23.919706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.070585 (+2150879us) service_pool.cc:225] Handling call
0114 08:59:26.093063 (+ 22478us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.093597 (+ 534us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.093625 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:26.094339 (+ 714us) log.cc:844] Serialized 26 byte log entry
0114 08:59:26.172733 (+ 78394us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:26.352778 (+180045us) log.cc:844] Serialized 14 byte log entry
0114 08:59:26.353059 (+ 281us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.353139 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:26.452033 (+ 98894us) raft_consensus.cc:1680] finished
0114 08:59:26.452053 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.452174 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":251248,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":3445,"raft.queue_time_us":23433,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":10878,"spinlock_wait_cycles":6397696,"thread_start_us":181329,"threads_started":2,"wal-append.queue_time_us":261931}
I20250114 08:59:26.459383 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.469477 21518 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:26.448212 24054 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:26.480412 21548 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:26.460933 24033 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:26.488924 24033 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.489370 24033 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:26.490195 21668 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.491261 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.492059 21702 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
W20250114 08:59:26.493119 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 59270.2us
I20250114 08:59:26.493485 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.494539 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1536) took 2661 ms. Trace:
I20250114 08:59:26.494747 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.495528 21554 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:26.495098 21627 rpcz_store.cc:276] 0114 08:59:23.833363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.079073 (+2245710us) service_pool.cc:225] Handling call
0114 08:59:26.109262 (+ 30189us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.109865 (+ 603us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.109903 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:26.110642 (+ 739us) log.cc:844] Serialized 26 byte log entry
0114 08:59:26.147603 (+ 36961us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.147640 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.147707 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:26.494288 (+346581us) raft_consensus.cc:1680] finished
0114 08:59:26.494311 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.494450 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":36348,"threads_started":1,"wal-append.queue_time_us":36825}
I20250114 08:59:26.496706 24074 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:26.497339 24074 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.498420 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1555) took 2588 ms. Trace:
I20250114 08:59:26.498899 21627 rpcz_store.cc:276] 0114 08:59:23.909766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.496913 (+2587147us) service_pool.cc:225] Handling call
0114 08:59:26.497297 (+ 384us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.498001 (+ 704us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.498032 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.498096 (+ 64us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.498173 (+ 77us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.498237 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.498354 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.499771 24074 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.500744 21627 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:26.465092 21623 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:26.477881 21550 raft_consensus.cc:3049] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:26.475135 22239 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.463280 21695 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:26.484059 24054 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.497674 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1549) took 2575 ms. Trace:
I20250114 08:59:26.501399 23903 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:26.521202 24054 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.521762 21550 raft_consensus.cc:738] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.519140 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:26.501443 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1561) took 2590 ms. Trace:
I20250114 08:59:26.522971 21627 rpcz_store.cc:276] 0114 08:59:23.911407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.500073 (+2588666us) service_pool.cc:225] Handling call
0114 08:59:26.500432 (+ 359us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.501200 (+ 768us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.501390 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.524664 21627 raft_consensus.cc:1270] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:26.526849 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1562) took 2615 ms. Trace:
I20250114 08:59:26.520735 21624 rpcz_store.cc:276] 0114 08:59:23.903706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.444830 (+2541124us) service_pool.cc:225] Handling call
0114 08:59:26.476356 (+ 31526us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.477293 (+ 937us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.477406 (+ 113us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.477569 (+ 163us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.477710 (+ 141us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.477937 (+ 227us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.478876 (+ 939us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.527303 24070 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.527614 21627 rpcz_store.cc:276] 0114 08:59:23.911669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.523782 (+2612113us) service_pool.cc:225] Handling call
0114 08:59:26.524219 (+ 437us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.526528 (+ 2309us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.526772 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.519680 23912 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.530851 23912 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.525698 21550 consensus_queue.cc:260] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.532060 21550 raft_consensus.cc:3054] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.494594 21521 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.534416 21622 raft_consensus.cc:1235] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:26.519999 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1547) took 2649 ms. Trace:
I20250114 08:59:26.535545 21521 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:26.535852 23801 raft_consensus.cc:3049] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:26.536643 21521 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.537285 21668 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.537930 24025 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:26.536427 23984 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:26.520506 24095 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.539602 23874 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:26.541850 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.540300 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.528012 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:26.542775 21629 raft_consensus.cc:2388] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:26.543037 21627 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:26.528203 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 54171.5us
I20250114 08:59:26.548755 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:26.549722 22267 raft_consensus.cc:2388] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.550171 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 380) took 2535 ms. Trace:
I20250114 08:59:26.542245 24096 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:26.543090 21624 raft_consensus.cc:2388] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:26.551741 24124 consensus_queue.cc:1046] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:26.552997 24124 raft_consensus.cc:3049] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:26.552945 23899 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:26.553478 24124 raft_consensus.cc:738] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.550367 24095 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.543936 23874 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:26.536362 23801 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.536005 21623 rpcz_store.cc:276] 0114 08:59:23.871221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.388113 (+2516892us) service_pool.cc:225] Handling call
0114 08:59:26.428363 (+ 40250us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.519532 (+ 91169us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.519873 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:26.556924 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 122432us
I20250114 08:59:26.557399 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:26.535154 21630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1563) took 2623 ms (client timeout 2235 ms). Trace:
W20250114 08:59:26.558395 21630 rpcz_store.cc:269] 0114 08:59:23.911952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.534748 (+2622796us) service_pool.cc:212] Skipping call since client already timed out
0114 08:59:26.534943 (+ 195us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 08:59:26.551514 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:26.559368 21624 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:26.559465 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.560384 21630 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:26.551426 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1739) took 2083 ms. Trace:
I20250114 08:59:26.556349 23874 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:26.562482 24071 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:26.562904 24129 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.561980 21696 rpcz_store.cc:276] 0114 08:59:24.467752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.173927 (+1706175us) service_pool.cc:225] Handling call
0114 08:59:26.416289 (+242362us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.432982 (+ 16693us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.433013 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.433069 (+ 56us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.433109 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.433166 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.551268 (+118102us) spinlock_profiling.cc:243] Waited 88.3 ms on lock 0x7b5400003a50. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.551368 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":203752576}
I20250114 08:59:26.563241 24071 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.566642 23972 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.004s
I20250114 08:59:26.563498 24129 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.525346 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.568037 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1554) took 2658 ms. Trace:
I20250114 08:59:26.455755 24000 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:26.566766 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:26.568818 24000 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:26.569306 24000 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:26.570019 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1493) took 2656 ms. Trace:
I20250114 08:59:26.570394 21551 rpcz_store.cc:276] 0114 08:59:23.913781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.913937 (+ 156us) service_pool.cc:225] Handling call
0114 08:59:23.914258 (+ 321us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:24.477962 (+563704us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:24.477994 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:24.478031 (+ 37us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:24.478069 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.569614 (+2091545us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b540031b354. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:26.569680 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.569831 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1295913728}
I20250114 08:59:26.562927 23874 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.572953 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:26.573369 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:26.572953 23874 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.576977 22270 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20250114 08:59:26.583014 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 107443us
I20250114 08:59:26.558599 21623 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:26.545879 21701 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:26.570953 24129 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.554411 24124 consensus_queue.cc:260] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.589283 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:26.589715 24124 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:26.590477 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.587719 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:26.556336 23801 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
W20250114 08:59:26.534184 21625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1548) took 2651 ms (client timeout 1845 ms). Trace:
I20250114 08:59:26.564639 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:26.618764 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 73374.7us
W20250114 08:59:26.620865 21625 rpcz_store.cc:269] 0114 08:59:23.882805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:23.883016 (+ 211us) service_pool.cc:225] Handling call
0114 08:59:26.533840 (+2650824us) spinlock_profiling.cc:243] Waited 62.1 ms on lock 0x7b54001518d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.533999 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142815488}
I20250114 08:59:26.535209 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1530) took 2708 ms. Trace:
I20250114 08:59:26.565496 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:26.565778 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:26.550498 22267 rpcz_store.cc:276] 0114 08:59:24.015159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.548218 (+2533059us) service_pool.cc:225] Handling call
0114 08:59:26.550089 (+ 1871us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.621130 21696 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:26.551142 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:26.621779 23801 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:26.627127 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.622121 21622 rpcz_store.cc:276] 0114 08:59:23.827051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:25.981024 (+2153973us) service_pool.cc:225] Handling call
0114 08:59:26.533986 (+552962us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800002d98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:26.534218 (+ 232us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.534909 (+ 691us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.535129 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28959616}
I20250114 08:59:26.460739 23913 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.629055 24070 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:26.625290 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.629693 21704 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:26.630286 21549 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.629833 24070 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.622992 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1564) took 2710 ms. Trace:
I20250114 08:59:26.631274 21625 rpcz_store.cc:276] 0114 08:59:23.912226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.621998 (+2709772us) service_pool.cc:225] Handling call
0114 08:59:26.622390 (+ 392us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.622592 (+ 202us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.622618 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.622680 (+ 62us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:26.622724 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.622795 (+ 71us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.622910 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.632143 24070 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.632431 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.626646 21696 raft_consensus.cc:1270] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250114 08:59:26.626375 21627 raft_consensus.cc:2388] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:26.633519 21696 pending_rounds.cc:77] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 3
I20250114 08:59:26.634136 21696 pending_rounds.cc:99] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250114 08:59:26.634478 21696 raft_consensus.cc:2883] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:26.635277 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 233) took 2722 ms. Trace:
I20250114 08:59:26.586865 21624 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:26.539006 23984 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:26.635917 23984 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:26.636355 23984 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.568440 21626 rpcz_store.cc:276] 0114 08:59:23.909464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.470817 (+2561353us) service_pool.cc:225] Handling call
0114 08:59:26.484941 (+ 14124us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.526129 (+ 41188us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.526163 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.526209 (+ 46us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:26.526270 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.567803 (+ 41533us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54001658d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:26.567844 (+ 41us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.567974 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94120704}
I20250114 08:59:26.592406 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:26.592681 24082 consensus_queue.cc:1159] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" responder_term: 7 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20250114 08:59:26.629194 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.638273 21696 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:26.638438 21622 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:26.639011 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.640022 21593 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:26.640704 23972 consensus_queue.cc:1046] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:59:26.642024 24129 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.642073 24074 raft_consensus.cc:3049] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:26.642632 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 235) took 2305 ms. Trace:
I20250114 08:59:26.642716 24074 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.643083 21626 rpcz_store.cc:276] 0114 08:59:24.337700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.637459 (+2299759us) service_pool.cc:225] Handling call
0114 08:59:26.637836 (+ 377us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.642324 (+ 4488us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.642348 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.642383 (+ 35us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:26.642418 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.642470 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.642578 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.643863 24074 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.645393 24074 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.646126 21626 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250114 08:59:26.646978 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.647296 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1750) took 2173 ms. Trace:
I20250114 08:59:26.647656 21704 rpcz_store.cc:276] 0114 08:59:24.474247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.350445 (+1876198us) service_pool.cc:225] Handling call
0114 08:59:26.366085 (+ 15640us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.630210 (+264125us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.647088 (+ 16878us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b540002f154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:26.647240 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":605775744}
I20250114 08:59:26.592412 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.598157 24094 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:26.649148 21704 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250114 08:59:26.454190 24116 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.543924 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:26.652840 21699 raft_consensus.cc:1235] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:26.655507 24116 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.606958 22270 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:26.593076 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 50004.6us
I20250114 08:59:26.613963 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.616397 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.616724 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.618202 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.618299 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.619903 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.591367 21701 raft_consensus.cc:2405] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 11 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:26.636914 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 234) took 2637 ms. Trace:
I20250114 08:59:26.637009 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.555001 24053 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.599596 24071 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.651402 21704 raft_consensus.cc:1235] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:26.600515 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 50598.3us
I20250114 08:59:26.663388 21554 raft_consensus.cc:2405] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20250114 08:59:26.603497 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 74) took 6825 ms. Trace:
I20250114 08:59:26.662319 21624 rpcz_store.cc:276] 0114 08:59:23.999582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.636014 (+2636432us) service_pool.cc:225] Handling call
0114 08:59:26.636314 (+ 300us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.636557 (+ 243us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.636585 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.636638 (+ 53us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:26.636682 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.636742 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.636856 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.493166 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:26.655851 21550 raft_consensus.cc:1270] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:26.633486 21695 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [8.3-8.3]
I20250114 08:59:26.661209 21625 consensus_queue.cc:237] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.685442 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.686544 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.715852 21549 raft_consensus.cc:2388] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.716255 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.716931 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.698932 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1386) took 2408 ms. Trace:
I20250114 08:59:26.718290 21550 rpcz_store.cc:276] 0114 08:59:24.290709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.479433 (+188724us) service_pool.cc:225] Handling call
0114 08:59:24.556679 (+ 77246us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.477627 (+1920948us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540009efa0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f55d990 00007fa63f5cf300 00007fa63f5d0371 00007fa63f5ccedf 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc
0114 08:59:26.675821 (+198194us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.698853 (+ 23032us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3236036480}
W20250114 08:59:26.685858 23781 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
W20250114 08:59:26.713171 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 76710us
I20250114 08:59:26.683951 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.683094 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.684227 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.685179 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.717721 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.635664 21627 rpcz_store.cc:276] 0114 08:59:23.913302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.634352 (+2721050us) service_pool.cc:225] Handling call
0114 08:59:26.634705 (+ 353us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.634967 (+ 262us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.634990 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.635024 (+ 34us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:26.635067 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.635124 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.635220 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:26.698694 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.726171 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:26.734894 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 48128.9us
I20250114 08:59:26.734544 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1747) took 2261 ms. Trace:
I20250114 08:59:26.737380 21521 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:26.684111 24116 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.738843 24129 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.684406 24095 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.685902 24033 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.739861 24095 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.741712 24033 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.741765 23984 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.743939 21700 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:26.744019 23984 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.725044 21702 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:26.747076 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1753) took 2272 ms. Trace:
I20250114 08:59:26.747416 21700 rpcz_store.cc:276] 0114 08:59:24.475167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.392596 (+1917429us) service_pool.cc:225] Handling call
0114 08:59:26.436973 (+ 44377us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.744483 (+307510us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.746837 (+ 2354us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b5400129154. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:26.747013 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":705477504}
I20250114 08:59:26.748746 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.750020 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.753150 21521 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:26.754354 23972 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:26.754803 23972 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:26.755211 23972 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:26.756762 24094 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 08:59:26.756891 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.757894 24091 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:26.758261 24091 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.759301 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.759178 24091 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.760073 21553 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:26.754961 24075 consensus_queue.cc:1046] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:26.761196 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.761341 24075 raft_consensus.cc:3049] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:26.761895 24075 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.762199 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.763121 24075 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.766729 21552 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:26.736614 21696 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
W20250114 08:59:26.765276 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 56236us
I20250114 08:59:26.772845 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.777998 21546 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:26.774471 21702 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:26.685935 24053 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250114 08:59:26.785789 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 313505us
I20250114 08:59:26.787083 22239 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:26.755208 23899 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:26.789870 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.741715 24095 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.796111 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.799294 21546 raft_consensus.cc:2405] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:26.796452 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.780797 24075 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:26.800108 21551 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:26.775130 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.794414 23899 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:26.808945 23899 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:26.664491 21451 rpcz_store.cc:276] 0114 08:59:19.778004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:19.778182 (+ 178us) service_pool.cc:225] Handling call
0114 08:59:26.603240 (+6825058us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:22.019310 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:22.022212 (+ 2902us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:22.022260 (+ 48us) write_op.cc:435] Acquired schema lock
0114 08:59:22.022281 (+ 21us) tablet.cc:659] Decoding operations
0114 08:59:22.023461 (+ 1180us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:22.023545 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:22.023585 (+ 40us) tablet.cc:682] Acquiring locks for 8 operations
0114 08:59:22.590179 (+566594us) tablet.cc:698] Row locks acquired
0114 08:59:22.590205 (+ 26us) write_op.cc:260] PREPARE: finished
0114 08:59:22.749533 (+159328us) write_op.cc:270] Start()
0114 08:59:22.799723 (+ 50190us) write_op.cc:276] Timestamp: P: 1736845162722560 usec, L: 0
0114 08:59:22.799770 (+ 47us) op_driver.cc:348] REPLICATION: starting
0114 08:59:23.013448 (+213678us) log.cc:844] Serialized 3241 byte log entry
0114 08:59:23.741102 (+727654us) op_driver.cc:464] REPLICATION: finished
0114 08:59:23.977715 (+236613us) write_op.cc:301] APPLY: starting
0114 08:59:23.998816 (+ 21101us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:24.100999 (+102183us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:24.137736 (+ 36737us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:24.138828 (+ 1092us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:24.139056 (+ 228us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0114 08:59:24.139099 (+ 43us) write_op.cc:312] APPLY: finished
0114 08:59:24.139381 (+ 282us) log.cc:844] Serialized 64 byte log entry
0114 08:59:24.139563 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:24.139704 (+ 141us) write_op.cc:454] Released schema lock
0114 08:59:24.139752 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:24.214836 (+ 75084us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:24.215792 (+ 956us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8575,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":210663,"apply.run_cpu_time_us":3381,"apply.run_wall_time_us":188373,"num_ops":8,"prepare.queue_time_us":400818,"prepare.run_cpu_time_us":23715,"prepare.run_wall_time_us":1278519,"raft.queue_time_us":289053,"raft.run_cpu_time_us":13589,"raft.run_wall_time_us":146740,"replication_time_us":927925,"thread_start_us":569931,"threads_started":4,"wal-append.queue_time_us":365346,"wal-append.run_cpu_time_us":18233,"wal-append.run_wall_time_us":1795505}]]}
I20250114 08:59:26.810523 21554 raft_consensus.cc:2388] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.814150 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.815066 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.801833 23935 raft_consensus.cc:2743] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.819476 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.795708 24124 consensus_queue.cc:1035] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20250114 08:59:26.820992 23899 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.738871 24116 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.737255 21698 rpcz_store.cc:276] 0114 08:59:24.473340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.214352 (+1741012us) service_pool.cc:225] Handling call
0114 08:59:26.379482 (+165130us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b5800297398. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:26.379889 (+ 407us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:26.389395 (+ 9506us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.389442 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:26.395230 (+ 5788us) log.cc:844] Serialized 52 byte log entry
0114 08:59:26.661802 (+266572us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:26.687635 (+ 25833us) log.cc:844] Serialized 14 byte log entry
0114 08:59:26.688101 (+ 466us) log.cc:844] Serialized 14 byte log entry
0114 08:59:26.705600 (+ 17499us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.705709 (+ 109us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:26.734266 (+ 28557us) raft_consensus.cc:1680] finished
0114 08:59:26.734293 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.734467 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17012,"prepare.queue_time_us":43166,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":14560,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":46636,"spinlock_wait_cycles":43469824,"thread_start_us":232059,"threads_started":1,"wal-append.queue_time_us":251513}
I20250114 08:59:26.823665 24116 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.824743 24123 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:26.825295 24123 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.826207 21549 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:26.826846 21547 raft_consensus.cc:2405] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:26.827890 23899 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:26.831629 21547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1503) took 2837 ms (client timeout 2709 ms). Trace:
W20250114 08:59:26.832193 21547 rpcz_store.cc:269] 0114 08:59:23.993801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:24.000884 (+ 7083us) service_pool.cc:225] Handling call
0114 08:59:26.831328 (+2830444us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b540007b2d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.831502 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5673740672}
I20250114 08:59:26.833045 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.833557 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.834196 21704 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:26.835856 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.836177 23972 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:26.838033 24091 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:26.842497 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1381) took 2920 ms. Trace:
I20250114 08:59:26.859491 21596 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:26.862072 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.863044 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.794302 23801 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20250114 08:59:26.862244 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.851483 23913 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:26.843138 21697 rpcz_store.cc:276] 0114 08:59:23.921603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.065506 (+2143903us) service_pool.cc:225] Handling call
0114 08:59:26.415105 (+349599us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.431245 (+ 16140us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.431278 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:26.431889 (+ 611us) log.cc:844] Serialized 26 byte log entry
0114 08:59:26.665233 (+233344us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.764730 (+ 99497us) log.cc:844] Serialized 14 byte log entry
0114 08:59:26.840836 (+ 76106us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b5400387820. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f561213 00007fa63f5cdf0f 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc
0114 08:59:26.840902 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.841005 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:26.841067 (+ 62us) raft_consensus.cc:1680] finished
0114 08:59:26.841197 (+ 130us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.842251 (+ 1054us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400370fd0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:26.842391 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43199,"prepare.queue_time_us":104926,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":16619,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":6315,"spinlock_wait_cycles":741394560,"thread_start_us":232876,"threads_started":1,"wal-append.queue_time_us":227192}
I20250114 08:59:26.870663 21594 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:26.861855 24116 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:26.904129 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 59386.7us
I20250114 08:59:26.904006 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.906790 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:26.871104 24129 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:26.909389 24129 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:26.909929 24129 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:26.910533 24093 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20250114 08:59:26.901332 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.913110 21698 pending_rounds.cc:77] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 2
I20250114 08:59:26.913453 21698 pending_rounds.cc:99] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 7.3
I20250114 08:59:26.913815 21698 raft_consensus.cc:2883] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:26.913506 24125 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 6.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:26.914319 24124 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:26.854269 24000 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.916782 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.917646 21549 raft_consensus.cc:2388] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.918418 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.904006 23781 consensus_queue.cc:1046] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20250114 08:59:26.920580 24116 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:26.921451 21453 catalog_manager.cc:5526] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:26.923367 21554 pending_rounds.cc:77] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 1
I20250114 08:59:26.923787 21554 pending_rounds.cc:99] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:26.924333 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.925187 21552 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:26.925304 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.925931 21555 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:26.926245 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.926810 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.927409 21555 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:26.927441 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.928929 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.930001 21696 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:26.930748 24123 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.931223 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:26.931365 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.932241 21696 raft_consensus.cc:2405] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20250114 08:59:26.933709 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.924407 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.934876 21704 raft_consensus.cc:2405] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:26.901324 24070 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:26.936935 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.937708 21701 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:26.938741 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.859143 23973 consensus_queue.cc:1046] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250114 08:59:26.924073 21554 raft_consensus.cc:2883] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:26.943804 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.944084 24125 raft_consensus.cc:3049] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:26.944487 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:26.913179 23972 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:26.941553 21596 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:26.950016 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.950207 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1745) took 2479 ms. Trace:
I20250114 08:59:26.950798 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.947794 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.944608 24125 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.953776 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.918345 24000 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:26.958070 21700 raft_consensus.cc:2388] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:26.950649 21703 rpcz_store.cc:276] 0114 08:59:24.471086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.201376 (+1730290us) service_pool.cc:225] Handling call
0114 08:59:26.201843 (+ 467us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:26.202286 (+ 443us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:26.402354 (+200068us) log.cc:844] Serialized 14 byte log entry
0114 08:59:26.946763 (+544409us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:26.946819 (+ 56us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.949893 (+ 3074us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.950035 (+ 142us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:26.950148 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46845,"prepare.queue_time_us":256591,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":72,"raft.queue_time_us":803,"thread_start_us":542818,"threads_started":2}
I20250114 08:59:26.961256 21521 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:26.935999 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.943995 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.956925 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.940045 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.969631 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.970422 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.954686 23903 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:26.972234 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.974769 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:26.976300 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:26.977407 21703 raft_consensus.cc:2371] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6. Re-sending same reply.
I20250114 08:59:26.974366 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:26.987280 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:26.987846 21700 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:26.988054 21696 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:26.967070 24070 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:26.991815 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:26.924028 21453 catalog_manager.cc:5526] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 6, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 6 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:26.996572 21704 raft_consensus.cc:2388] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 4: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:26.997437 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.998049 21627 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:26.988793 24000 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:26.998862 24000 raft_consensus.cc:695] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:26.999311 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:26.999845 24000 consensus_queue.cc:237] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.001221 21553 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
W20250114 08:59:27.001432 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 59981.5us
I20250114 08:59:27.001260 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.994014 21594 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:27.005898 24062 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:27.006049 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:27.007140 21703 raft_consensus.cc:2388] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:26.986217 23781 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250114 08:59:26.986012 24124 raft_consensus.cc:3049] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:27.017359 24124 raft_consensus.cc:738] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:27.026526 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.027822 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:27.028383 21704 raft_consensus.cc:2371] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7. Re-sending same reply.
I20250114 08:59:27.029496 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.029163 24125 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.031078 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.031244 21596 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.032294 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.028851 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.034181 21701 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:27.038990 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.036852 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:27.042237 21704 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:27.043612 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:26.991962 24070 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.044514 21701 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:27.044682 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.050998 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:27.018651 24124 consensus_queue.cc:260] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:27.054183 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 47634.9us
I20250114 08:59:27.010668 24116 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:27.055555 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.048888 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.056350 21695 raft_consensus.cc:2388] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:27.056744 21704 raft_consensus.cc:2388] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 5: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:27.055583 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.046247 24025 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.066839 21551 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250114 08:59:27.048800 24070 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:27.064756 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 70816.6us
I20250114 08:59:27.064496 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.066197 21453 catalog_manager.cc:5526] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 4, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.043962 24095 consensus_queue.cc:1035] T 270ce1b254bd4b7180e48e18af79cda6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250114 08:59:27.094370 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:27.106108 21697 raft_consensus.cc:1235] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:27.108511 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.109724 21703 raft_consensus.cc:2388] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:27.110762 21704 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:27.112015 21701 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 08:59:27.083772 23784 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.118722 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.127636 21695 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
W20250114 08:59:27.123216 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 68047.5us
I20250114 08:59:27.128033 21695 pending_rounds.cc:77] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 3
I20250114 08:59:27.128412 21695 pending_rounds.cc:99] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250114 08:59:27.128676 21695 raft_consensus.cc:2883] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:27.125113 21594 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:27.135268 21669 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.096809 24025 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.127789 21700 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:27.098166 24062 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:27.127926 21453 catalog_manager.cc:5526] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 4, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 4 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:27.173112 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.181113 21669 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.139158 24124 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:27.186524 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:27.173099 23472 raft_consensus.cc:2758] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250114 08:59:27.198558 24116 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:27.198705 23824 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.199708 23824 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:27.200510 22241 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:27.201800 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 65923.1us
I20250114 08:59:27.202699 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.204559 21453 catalog_manager.cc:5526] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 4, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } }
I20250114 08:59:27.206897 21453 catalog_manager.cc:5526] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 5, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
W20250114 08:59:27.217257 24145 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250114 08:59:27.222536 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57308 (request call id 477) took 3308 ms. Trace:
I20250114 08:59:27.241922 21628 rpcz_store.cc:276] 0114 08:59:23.915639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.650222 (+2734583us) service_pool.cc:225] Handling call
0114 08:59:27.221594 (+571372us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54003b9cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f552d50 00007fa63f5d5f2a 00007fa64452fc5e 00007fa63dc0c817 00007fa63dc0c7a2 00007fa63dc0c6d1 00007fa63dc0c64c 00007fa63dc0b902 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:27.222400 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":956368768}
W20250114 08:59:27.230794 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 111170us
I20250114 08:59:27.245610 21668 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:27.246752 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:27.234720 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 230) took 3342 ms. Trace:
I20250114 08:59:27.235600 23935 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:27.226456 23702 raft_consensus.cc:3049] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:27.257154 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:27.139516 24129 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:27.275646 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 78303.2us
I20250114 08:59:27.247781 21621 rpcz_store.cc:276] 0114 08:59:23.892135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.438339 (+2546204us) service_pool.cc:225] Handling call
0114 08:59:26.496990 (+ 58651us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.497526 (+ 536us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.497556 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:26.498083 (+ 527us) log.cc:844] Serialized 26 byte log entry
0114 08:59:26.980148 (+482065us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:26.980206 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:26.980297 (+ 91us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:27.234424 (+254127us) raft_consensus.cc:1680] finished
0114 08:59:27.234450 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:27.234643 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1236,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":481638,"threads_started":1,"wal-append.queue_time_us":627932}
I20250114 08:59:27.250023 21668 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:27.282328 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 101311us
I20250114 08:59:27.283336 21669 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:27.273068 23824 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:27.287951 23824 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:27.288445 23824 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.270609 24116 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:27.294258 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:27.299347 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 49399us
I20250114 08:59:27.295814 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:27.272173 23702 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:27.295163 21669 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:27.300627 21704 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:27.290575 24062 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:27.304729 24095 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:27.323297 21453 catalog_manager.cc:5526] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.330766 21695 raft_consensus.cc:1270] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:27.300942 24116 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:27.306365 23472 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250114 08:59:27.352084 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 57013.6us
I20250114 08:59:27.325222 24095 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:27.324082 24062 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:27.341809 24116 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:27.374348 21521 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:27.343525 23472 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:27.383622 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 352425us
I20250114 08:59:27.377295 21453 catalog_manager.cc:5526] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 4, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.321094 23935 consensus_queue.cc:1046] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20250114 08:59:27.368641 24095 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.370224 24062 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:27.350258 24123 consensus_queue.cc:1046] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20250114 08:59:27.406227 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250114 08:59:27.406870 24147 log.cc:926] Time spent T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20250114 08:59:27.371035 23781 consensus_queue.cc:1046] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
W20250114 08:59:27.401167 24148 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.113s user 0.002s sys 0.000s
I20250114 08:59:27.391853 24093 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:27.388343 23472 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:27.387534 24053 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
W20250114 08:59:27.424448 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 299530us
I20250114 08:59:27.405841 21453 catalog_manager.cc:5526] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 4, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 4 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.445943 22270 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250114 08:59:27.444754 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 167744us
I20250114 08:59:27.404879 23984 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20250114 08:59:27.455092 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 252131us
I20250114 08:59:27.417111 24093 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:27.462989 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 71492.1us
I20250114 08:59:27.426602 24129 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:27.436012 24033 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:27.486351 21699 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:27.471230 21453 catalog_manager.cc:5526] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.408983 23702 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.493362 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:27.474313 24093 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:27.510775 21696 consensus_meta.cc:287] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.503s user 0.011s sys 0.002s
I20250114 08:59:27.511822 21696 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:27.516225 21703 raft_consensus.cc:1235] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:27.522436 24164 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 08:59:27.524963 24164 consensus_queue.cc:1225] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20250114 08:59:27.525664 24149 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20250114 08:59:27.524986 24163 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:27.527346 24163 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:27.527771 24163 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:27.528656 24153 log.cc:926] Time spent T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20250114 08:59:27.537114 23702 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:27.515031 24071 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250114 08:59:27.544778 24075 raft_consensus.cc:3049] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:27.545341 24075 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:27.552703 24169 raft_consensus.cc:3049] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:27.553037 23984 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:27.553206 24169 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:27.553476 23984 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:27.553890 23984 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:27.554172 24169 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:27.555583 24169 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:27.562279 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 143065us
W20250114 08:59:27.564059 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 138624us
W20250114 08:59:27.592095 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 84220.6us
I20250114 08:59:27.597506 21596 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.602489 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:27.602967 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 147868us
I20250114 08:59:27.594600 21453 catalog_manager.cc:5526] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 6, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 6 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
W20250114 08:59:27.604429 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 60931us
I20250114 08:59:27.605020 22241 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.546554 24075 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.554023 24030 consensus_queue.cc:1035] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.058s
W20250114 08:59:27.612444 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 106348us
I20250114 08:59:27.620079 21453 catalog_manager.cc:5526] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 6, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 6 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
W20250114 08:59:27.624131 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 164252us
I20250114 08:59:27.617470 24129 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.620419 23913 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:27.631258 24181 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.620048 23903 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.632848 24175 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:27.635526 23973 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:27.637730 24033 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.610518 24082 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250114 08:59:27.653090 24165 consensus_queue.cc:1046] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20250114 08:59:27.644526 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:27.655074 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57680.2us
I20250114 08:59:27.655186 21453 catalog_manager.cc:5526] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 5, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
W20250114 08:59:27.629698 24151 log.cc:926] Time spent T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
I20250114 08:59:27.645357 23935 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:27.649466 23472 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:27.696734 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 68531.5us
I20250114 08:59:27.690420 21596 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:27.673969 23973 consensus_queue.cc:1225] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:27.677083 24033 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.636320 24095 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:27.676425 23903 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:27.701932 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 72822.9us
I20250114 08:59:27.654347 24162 consensus_queue.cc:1046] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20250114 08:59:27.708245 21695 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:27.705835 21453 catalog_manager.cc:5526] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 5, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.709889 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:27.656849 23913 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.702265 22271 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250114 08:59:27.718091 24025 consensus_meta.cc:287] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.567s user 0.012s sys 0.000s
I20250114 08:59:27.718545 24025 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.724188 21669 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.726208 21621 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:27.706386 24129 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:27.716454 21453 catalog_manager.cc:5526] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 4 to 6, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 6 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.728365 21453 catalog_manager.cc:5526] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 2, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 2 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.667671 24181 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:27.746863 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 141918us
I20250114 08:59:27.748703 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:27.749266 21596 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.750380 21553 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:27.713605 24167 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:27.751091 24167 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:27.751444 24167 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:27.751736 23913 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:27.697778 23824 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.755046 21669 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.756582 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.684156 24175 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:27.759553 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.721202 24062 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:27.740707 24129 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.753269 24174 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:27.744421 24025 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:27.770766 21453 catalog_manager.cc:5526] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 4, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 4 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.774027 22241 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:27.757189 24070 consensus_meta.cc:287] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.669s user 0.006s sys 0.006s
I20250114 08:59:27.782207 21696 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:27.784097 21669 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:27.791580 22241 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:27.796759 21596 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.785624 21453 catalog_manager.cc:5526] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 4, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 4 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:27.798177 21596 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.776829 24129 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:27.776542 23913 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:27.775013 24192 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.806577 21453 catalog_manager.cc:5526] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 4, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 4 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.779181 24025 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.785269 24162 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:27.781692 24070 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.788121 23972 consensus_queue.cc:1035] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20250114 08:59:27.827445 24175 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:27.824198 21453 catalog_manager.cc:5526] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 6, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 6 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.804277 24180 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.816144 23935 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:27.819026 24025 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:27.793290 21548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:27.837836 21669 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.839394 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:27.837242 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 49149.4us
W20250114 08:59:27.828493 24172 log.cc:926] Time spent T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.299s user 0.001s sys 0.000s
W20250114 08:59:27.847959 24157 log.cc:926] Time spent T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20250114 08:59:27.854519 24175 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.854810 24062 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:27.855082 21548 raft_consensus.cc:604] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.855563 21548 raft_consensus.cc:612] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:27.856359 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:27.857470 24191 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:27.858126 24191 raft_consensus.cc:695] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:27.859033 24191 consensus_queue.cc:237] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.827471 24183 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:27.813030 23801 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20250114 08:59:27.861575 23824 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:27.862707 24181 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:27.863554 24095 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:27.864800 23472 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:27.867170 21548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:27.867820 21548 raft_consensus.cc:604] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.868139 21548 raft_consensus.cc:612] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:27.818557 24189 consensus_queue.cc:1046] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250114 08:59:27.871208 24189 raft_consensus.cc:3049] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:27.871552 24189 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:27.891198 21453 catalog_manager.cc:5526] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 5, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
W20250114 08:59:27.894042 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 51941.3us
W20250114 08:59:27.852568 24158 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
I20250114 08:59:27.893604 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:27.872349 24189 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.900024 24189 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:27.903192 23702 consensus_queue.cc:1159] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250114 08:59:27.905270 24033 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.907438 24033 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:27.910064 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.910907 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45060 (request call id 155) took 1096 ms. Trace:
I20250114 08:59:27.911607 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:27.912142 24124 consensus_meta.cc:287] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.719s user 0.007s sys 0.005s
I20250114 08:59:27.912392 21553 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:27.913450 24075 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:27.914496 24116 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.521s user 0.010s sys 0.001s
I20250114 08:59:27.914947 24116 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.915915 23874 raft_consensus.cc:3049] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:27.915969 24163 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.911314 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.897621 21453 catalog_manager.cc:5526] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 5, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.919112 21546 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.911288 21702 rpcz_store.cc:276] 0114 08:59:26.814122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.853180 (+ 39058us) service_pool.cc:225] Handling call
0114 08:59:26.853658 (+ 478us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.892375 (+ 38717us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.892420 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:26.949875 (+ 57455us) log.cc:844] Serialized 26 byte log entry
0114 08:59:27.358387 (+408512us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:27.358427 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.358509 (+ 82us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:27.910706 (+552197us) raft_consensus.cc:1680] finished
0114 08:59:27.910727 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:27.910832 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11776,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":4707,"thread_start_us":375025,"threads_started":1,"wal-append.queue_time_us":519742}
I20250114 08:59:27.910058 21668 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:27.921321 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.841487 23935 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:27.927832 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 90116.4us
I20250114 08:59:27.928786 21669 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:27.928813 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 89548.4us
I20250114 08:59:27.950104 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.950817 21555 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:27.924233 21453 catalog_manager.cc:5526] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 4, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 4 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.957911 21596 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.961882 21553 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
W20250114 08:59:27.962071 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 48790.5us
I20250114 08:59:27.920991 23874 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:27.924777 24033 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:27.962527 21553 raft_consensus.cc:604] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:27.928606 23935 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:27.974855 24179 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20250114 08:59:27.903213 23984 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.975591 24025 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.977752 24116 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:27.957024 24183 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:27.984354 21546 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:27.985451 23874 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.985767 21453 catalog_manager.cc:5526] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 4, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131). New cstate: current_term: 4 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.986802 23874 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:27.987526 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:27.986930 21453 catalog_manager.cc:5526] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 3, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:27.988857 24175 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:27.988675 24183 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.988644 21453 catalog_manager.cc:5526] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 6, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 6 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.989830 24163 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:27.990433 21453 catalog_manager.cc:5526] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 3 to 5, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:27.991278 24183 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:27.992146 21453 catalog_manager.cc:5526] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 2 to 3, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
W20250114 08:59:27.992902 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 64185.2us
I20250114 08:59:27.994809 23984 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:27.994179 24198 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:27.994405 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 63611.2us
I20250114 08:59:27.995795 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:27.997373 21453 catalog_manager.cc:5526] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:27.996356 21596 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:28.000680 24163 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:27.968425 23913 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:28.003587 21596 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:28.004781 24070 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.005461 24163 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:28.006016 24163 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:28.006911 24163 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:27.981925 24093 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.010820 24033 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:28.011024 21596 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:28.013669 21668 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:28.015879 21596 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:28.016642 21546 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:28.020651 21555 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:28.022749 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:28.026230 21668 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:28.034951 21596 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:28.030843 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:28.036326 21695 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:28.039177 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:28.018054 24070 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:28.047103 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:28.047703 21555 raft_consensus.cc:604] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:28.047762 21596 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:28.048110 21555 raft_consensus.cc:612] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:28.017318 24095 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:28.043874 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.026036 24174 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:28.054924 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.055789 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.055881 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.057407 21625 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
W20250114 08:59:28.057816 21520 leader_election.cc:336] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.522s (SENT)
I20250114 08:59:28.060253 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.060575 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.060902 21626 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:28.061134 21622 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:28.041231 24116 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:28.040613 23984 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.059968 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.064983 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.078130 21629 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:28.046173 24071 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.083544 24071 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.084669 23984 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.066968 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.042300 24124 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:28.086560 21623 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:28.115806 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:28.050868 24183 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:28.120501 21695 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:28.132615 21696 raft_consensus.cc:2363] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:28.108014 21546 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:28.056317 24093 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.064432 23784 raft_consensus.cc:3049] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
W20250114 08:59:28.129089 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 81393.3us
I20250114 08:59:28.124114 24181 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:28.148066 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 107332us
I20250114 08:59:28.148774 21596 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:28.063894 24188 raft_consensus.cc:3049] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:28.063499 24171 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250114 08:59:28.153298 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 95521us
W20250114 08:59:28.163012 24123 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.008s sys 0.017s
I20250114 08:59:28.170877 21621 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:28.171772 21630 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:28.118536 24166 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:28.231628 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.232403 21630 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:28.233736 21696 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:28.116530 24174 raft_consensus.cc:695] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:28.133016 24116 raft_consensus.cc:695] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:28.245416 21520 leader_election.cc:336] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.547s (SENT)
I20250114 08:59:28.247496 21521 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:28.148487 24160 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:28.152088 24167 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.148222 23784 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:28.171572 24188 raft_consensus.cc:738] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:28.063966 24182 consensus_queue.cc:1046] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.135s
I20250114 08:59:28.255196 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:28.256891 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 108174us
I20250114 08:59:28.266849 21596 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:28.230464 24123 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.814s user 0.009s sys 0.017s
I20250114 08:59:28.259119 21520 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:28.149029 24204 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 08:59:28.265847 21625 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:28.249842 24175 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:28.061883 24095 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:28.265060 22269 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:28.270493 21521 leader_election.cc:400] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:28.291862 21521 leader_election.cc:403] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250114 08:59:28.132357 24124 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.253242 24181 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:28.252997 24206 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.288306 21627 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:28.290051 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.245918 24129 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:28.302512 21625 raft_consensus.cc:2388] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 5: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:28.314329 21628 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
W20250114 08:59:28.297511 23903 consensus_meta.cc:287] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.584s user 0.011s sys 0.000s
I20250114 08:59:28.293816 24025 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.291282 23984 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.299417 24082 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 08:59:28.349467 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 147563us
W20250114 08:59:28.353154 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 90554us
I20250114 08:59:28.346793 21628 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:28.349231 23702 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:28.366026 21625 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:28.297277 24175 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:28.360695 24071 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.357977 24188 consensus_queue.cc:260] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.371565 23899 consensus_queue.cc:1046] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 08:59:28.312619 24206 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.343518 23472 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:28.373507 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:28.345795 23903 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.357553 24174 consensus_queue.cc:237] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:28.390837 24218 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
W20250114 08:59:28.395380 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 107180us
I20250114 08:59:28.367566 24093 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:28.360630 24199 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:28.402983 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 50042.1us
I20250114 08:59:28.403698 21696 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:28.404887 21630 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:28.375416 24129 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:28.316315 23784 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.416167 21622 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:28.416441 21521 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:28.419365 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.424490 24199 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:28.425943 24156 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.911s user 0.007s sys 0.018s
I20250114 08:59:28.427698 24199 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:28.428179 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 243) took 1497 ms. Trace:
I20250114 08:59:28.428377 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:28.429949 24189 consensus_meta.cc:287] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.529s user 0.008s sys 0.001s
I20250114 08:59:28.415751 24082 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.429107 22267 rpcz_store.cc:276] 0114 08:59:26.930782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.932266 (+ 1484us) service_pool.cc:225] Handling call
0114 08:59:26.932679 (+ 413us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:26.933113 (+ 434us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.933145 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:26.933653 (+ 508us) log.cc:844] Serialized 26 byte log entry
0114 08:59:27.198432 (+264779us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:27.198478 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.198583 (+ 105us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:28.425213 (+1226630us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:28.427819 (+ 2606us) raft_consensus.cc:1680] finished
0114 08:59:28.427850 (+ 31us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:28.428090 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1195,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":134,"spinlock_wait_cycles":6130944,"thread_start_us":259257,"threads_started":1,"wal-append.queue_time_us":425057}
I20250114 08:59:28.431056 24082 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.349742 24116 consensus_queue.cc:237] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:28.433471 24082 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.434984 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.435497 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:28.415205 24188 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:28.414187 24213 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:28.435547 23874 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:28.407388 23472 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.487672 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.491458 21629 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
W20250114 08:59:28.493050 21520 leader_election.cc:336] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.845s (SENT)
I20250114 08:59:28.494494 21622 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:28.501605 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.422616 24093 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.502316 21629 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:28.492766 23702 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:28.469282 24167 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.508210 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.508791 21622 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:28.493814 23874 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:28.497725 24207 log.cc:926] Time spent T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
I20250114 08:59:28.513648 23702 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:28.515741 21623 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
W20250114 08:59:28.543298 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 50286.8us
I20250114 08:59:28.534498 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.578305 21627 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:28.531028 24219 log.cc:926] Time spent T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.216s user 0.001s sys 0.000s
I20250114 08:59:28.581370 23702 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:28.515749 24199 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:28.585700 24199 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:28.586594 24095 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:28.586493 24093 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.562438 21696 raft_consensus.cc:1235] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:28.652387 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:28.627168 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 52609.1us
I20250114 08:59:28.867720 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:28.721907 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:28.873463 21555 raft_consensus.cc:604] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Received request to transfer leadership to TS d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:28.869316 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 151544us
I20250114 08:59:28.870841 21695 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:28.771867 21622 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:28.578840 21626 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
W20250114 08:59:28.522869 24176 log.cc:926] Time spent T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.817s user 0.000s sys 0.004s
W20250114 08:59:28.524224 23935 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.514s user 0.009s sys 0.000s
I20250114 08:59:28.732430 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:28.871680 21520 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.518s (SENT) [suppressed 57 similar messages]
I20250114 08:59:28.879777 21551 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20250114 08:59:28.881021 21551 pending_rounds.cc:77] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 3
I20250114 08:59:28.881641 21551 pending_rounds.cc:99] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20250114 08:59:28.512796 24213 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:28.527030 24169 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:28.882390 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:28.882620 24169 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:28.883081 21702 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 3.
I20250114 08:59:28.524379 23874 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:28.881950 21551 raft_consensus.cc:2883] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:28.884601 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.886324 21629 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:28.889447 21622 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:28.542794 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:28.536036 24175 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:28.891439 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:28.891647 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:28.894275 23913 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.810537 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:28.897133 24093 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.901818 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:28.902298 24173 log.cc:926] Time spent T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 1.379s user 0.000s sys 0.004s
W20250114 08:59:28.918162 24154 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 1.403s user 0.000s sys 0.002s
I20250114 08:59:28.920233 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1385) took 2054 ms. Trace:
W20250114 08:59:28.921734 24161 log.cc:926] Time spent T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 1.406s user 0.000s sys 0.005s
I20250114 08:59:28.920629 21698 rpcz_store.cc:276] 0114 08:59:26.865922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.895950 (+ 30028us) service_pool.cc:225] Handling call
0114 08:59:26.912868 (+ 16918us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:26.914579 (+ 1711us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.914611 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:26.921079 (+ 6468us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b2c00042708. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63dfae741 00007fa63dfbdd72 00007fa63f609141 00007fa63f5cd58a 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782
0114 08:59:26.921608 (+ 529us) log.cc:844] Serialized 52 byte log entry
0114 08:59:27.270531 (+348923us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:27.309189 (+ 38658us) log.cc:844] Serialized 14 byte log entry
0114 08:59:27.336848 (+ 27659us) log.cc:844] Serialized 14 byte log entry
0114 08:59:27.555967 (+219119us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b18003313c0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c694914 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:27.773231 (+217264us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b18003313c0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c691305 00007fa63f5c53d0 00007fa63f5d7666 00007fa63f5e81d7 00007fa63f5e8158
0114 08:59:27.811274 (+ 38043us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.811378 (+ 104us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:28.917524 (+1106146us) spinlock_profiling.cc:243] Waited 7.56 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:28.919983 (+ 2459us) raft_consensus.cc:1680] finished
0114 08:59:28.920003 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:28.920169 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":266045,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":124410,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":9033,"spinlock_wait_cycles":45209216,"thread_start_us":386311,"threads_started":2,"wal-append.queue_time_us":476000}
I20250114 08:59:28.924561 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1408) took 2004 ms. Trace:
I20250114 08:59:28.924945 21554 rpcz_store.cc:276] 0114 08:59:26.920515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.922439 (+ 1924us) service_pool.cc:225] Handling call
0114 08:59:26.923000 (+ 561us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:26.989004 (+ 66004us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:26.989052 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:26.991759 (+ 2707us) log.cc:844] Serialized 52 byte log entry
0114 08:59:27.354840 (+363081us) raft_consensus.cc:1640] Marking committed up to 1
0114 08:59:27.354877 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.354941 (+ 64us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:28.921021 (+1566080us) spinlock_profiling.cc:243] Waited 6.23 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:28.924231 (+ 3210us) raft_consensus.cc:1680] finished
0114 08:59:28.924250 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:28.924478 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12638,"raft.queue_time_us":12806,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":14321536,"thread_start_us":310550,"threads_started":1,"wal-append.queue_time_us":445134}
I20250114 08:59:28.926543 24093 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.911523 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:28.906322 21550 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:28.881464 23903 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:28.897619 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:28.913715 21520 leader_election.cc:336] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.235s (SENT)
I20250114 08:59:28.932075 21622 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:28.914006 24220 log.cc:926] Time spent T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.598s user 0.002s sys 0.000s
I20250114 08:59:28.934736 24094 raft_consensus.cc:3049] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:28.935181 24094 raft_consensus.cc:738] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:28.909992 24074 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.936779 24074 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.938170 21625 raft_consensus.cc:1270] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:28.938834 24074 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.928727 24093 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.879374 23472 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:28.750283 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:28.954706 21628 consensus_meta.cc:287] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.599s user 0.005s sys 0.008s
I20250114 08:59:28.602636 24199 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:28.943573 21627 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:28.953243 21622 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:28.903016 24206 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.897159 24125 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:28.902601 24116 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.909901 24228 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.886870 23935 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:28.932368 24124 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:28.953650 24030 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.910549 23913 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.971887 22267 raft_consensus.cc:2388] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 4: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:29.006024 22274 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:28.936277 24094 consensus_queue.cc:260] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:29.009479 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 127256us
I20250114 08:59:28.883181 24213 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.018683 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1786) took 1828 ms. Trace:
I20250114 08:59:29.023079 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.024119 24094 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:29.027686 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.029341 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.033041 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.058531 21554 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:29.019299 21701 rpcz_store.cc:276] 0114 08:59:27.190541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:27.190896 (+ 355us) service_pool.cc:225] Handling call
0114 08:59:27.267095 (+ 76199us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:27.287511 (+ 20416us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:27.287559 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:27.311084 (+ 23525us) log.cc:844] Serialized 26 byte log entry
0114 08:59:27.494914 (+183830us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:27.494986 (+ 72us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.495116 (+ 130us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:28.505518 (+1010402us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:29.008677 (+503159us) raft_consensus.cc:1680] finished
0114 08:59:29.008700 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:29.018587 (+ 9887us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11436,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":1942,"spinlock_wait_cycles":16771456,"thread_start_us":151666,"threads_started":1,"wal-append.queue_time_us":329912}
W20250114 08:59:29.066576 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 47200.9us
I20250114 08:59:29.079938 21550 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:29.080948 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.081208 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.081727 21555 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:29.082507 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.081758 21554 raft_consensus.cc:2363] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:29.014070 23972 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:28.972808 24116 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.088702 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:29.090780 24231 log.cc:926] Time spent T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250114 08:59:29.082607 24189 raft_consensus.cc:3049] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:29.093566 24189 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:29.094497 24189 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.095600 24189 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:29.096385 24116 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:29.096848 23824 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:29.100929 23801 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:29.101500 23801 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.977430 24228 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:28.538748 24186 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:29.107131 24169 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.112303 21551 raft_consensus.cc:2405] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20250114 08:59:29.113677 24230 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20250114 08:59:29.005123 24030 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:29.115689 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 848883us
I20250114 08:59:29.116417 21596 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.116922 24030 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.113421 21547 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:29.117589 21596 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:29.120373 24222 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250114 08:59:29.120940 24233 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20250114 08:59:29.121937 23913 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:29.126874 24188 consensus_meta.cc:287] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.657s user 0.008s sys 0.004s
W20250114 08:59:29.127123 24205 log.cc:926] Time spent T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 1.154s user 0.003s sys 0.011s
W20250114 08:59:29.127703 24221 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.638s user 0.005s sys 0.000s
I20250114 08:59:29.130156 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1430) took 1143 ms. Trace:
I20250114 08:59:29.130273 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.130582 21546 rpcz_store.cc:276] 0114 08:59:27.987120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:27.987273 (+ 153us) service_pool.cc:225] Handling call
0114 08:59:29.129999 (+1142726us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b540028e1d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:29.130096 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65252608}
I20250114 08:59:29.131444 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.132040 21695 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:29.132221 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.132754 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1376) took 1087 ms. Trace:
I20250114 08:59:29.132881 21696 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:29.133157 21623 rpcz_store.cc:276] 0114 08:59:28.044990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:28.045102 (+ 112us) service_pool.cc:225] Handling call
0114 08:59:29.132570 (+1087468us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b54000c7e54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:29.132688 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71354240}
I20250114 08:59:29.133859 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.134467 21699 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:29.124790 23874 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.135596 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.136185 21701 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:29.136622 21668 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:29.137559 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:29.138523 21546 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:29.138618 21547 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:29.144326 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1805) took 1328 ms. Trace:
I20250114 08:59:29.117626 23784 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:29.152083 24204 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:29.153374 24175 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:29.153486 23874 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.154057 24175 raft_consensus.cc:695] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:29.154974 24175 consensus_queue.cc:237] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.157761 24123 raft_consensus.cc:3049] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:29.158196 24123 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:29.159044 24123 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.160140 24123 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.161206 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.162602 24129 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:29.163203 24129 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:29.162667 24200 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:29.164744 24222 raft_consensus.cc:3049] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:29.165175 24222 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:29.165587 21700 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:29.165959 24222 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.166441 21521 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:29.167277 24222 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.168907 24228 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:29.169557 24168 log.cc:926] Time spent T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 1.639s user 0.000s sys 0.007s
W20250114 08:59:29.169843 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 275247us
I20250114 08:59:29.170970 21521 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:29.171877 21521 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:29.172128 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1780) took 2079 ms. Trace:
W20250114 08:59:29.172608 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 58952.1us
I20250114 08:59:29.164163 24129 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.172554 21697 rpcz_store.cc:276] 0114 08:59:27.092729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:27.107266 (+ 14537us) service_pool.cc:225] Handling call
0114 08:59:27.126508 (+ 19242us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:27.127042 (+ 534us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:27.127082 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:27.134607 (+ 7525us) log.cc:844] Serialized 52 byte log entry
0114 08:59:27.429683 (+295076us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:27.459642 (+ 29959us) log.cc:844] Serialized 14 byte log entry
0114 08:59:27.467537 (+ 7895us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.467820 (+ 283us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:29.168770 (+1700950us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:29.171813 (+ 3043us) raft_consensus.cc:1680] finished
0114 08:59:29.171917 (+ 104us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:29.172069 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7109,"prepare.queue_time_us":31671,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":3607,"raft.queue_time_us":150658,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":15280,"spinlock_wait_cycles":3827968,"thread_start_us":237643,"threads_started":1,"wal-append.queue_time_us":347027}
I20250114 08:59:29.180958 21521 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:29.147029 21703 rpcz_store.cc:276] 0114 08:59:27.815718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:27.816023 (+ 305us) service_pool.cc:225] Handling call
0114 08:59:27.824160 (+ 8137us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:27.840415 (+ 16255us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:27.840489 (+ 74us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:27.870753 (+ 30264us) log.cc:844] Serialized 52 byte log entry
0114 08:59:27.924677 (+ 53924us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:27.967484 (+ 42807us) log.cc:844] Serialized 14 byte log entry
0114 08:59:27.967670 (+ 186us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.967727 (+ 57us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:28.067581 (+ 99854us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b180012c6a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:28.081957 (+ 14376us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b180012c6a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c694914 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:29.126362 (+1044405us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:29.144095 (+ 17733us) raft_consensus.cc:1680] finished
0114 08:59:29.144118 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:29.144260 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51196,"prepare.queue_time_us":42652,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":10905,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":47193,"spinlock_wait_cycles":24013184,"thread_start_us":52930,"threads_started":1,"wal-append.queue_time_us":101302}
W20250114 08:59:29.172899 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 91510.1us
I20250114 08:59:29.169674 24178 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:29.212194 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.212766 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:59:29.213665 24167 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:29.214213 24167 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:29.215154 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:29.211890 24178 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:29.217118 24178 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:29.211825 24166 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:29.219405 21594 leader_election.cc:400] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:29.219744 21594 leader_election.cc:403] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 08:59:29.195762 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.193071 23874 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:29.193349 24160 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20250114 08:59:29.193508 24075 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:29.197788 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.199720 24082 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 08:59:29.200686 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 63193us
I20250114 08:59:29.230800 21596 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.230072 21555 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:29.205550 24181 consensus_queue.cc:1046] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:29.207064 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1371) took 1163 ms. Trace:
I20250114 08:59:29.219877 22274 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:29.233803 21621 rpcz_store.cc:276] 0114 08:59:28.043511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:28.043624 (+ 113us) service_pool.cc:225] Handling call
0114 08:59:29.206876 (+1163252us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400151dd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:29.206990 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275774336}
I20250114 08:59:29.208549 21630 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:29.208544 24183 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:29.210954 24184 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:29.216141 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:29.231773 24082 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:29.240902 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.241695 21696 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:29.220319 24071 consensus_queue.cc:1046] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20250114 08:59:29.241909 23903 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250114 08:59:29.249773 21596 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:29.250209 23935 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.250816 24025 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:29.251448 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.252535 24071 raft_consensus.cc:3049] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:29.252979 24071 raft_consensus.cc:738] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:29.253404 21701 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:29.253728 24071 consensus_queue.cc:260] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.232427 24075 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:29.255362 24075 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:29.256259 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.225869 23874 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:29.257135 24188 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:29.257417 23874 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.257548 24188 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:29.258036 24188 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:29.227200 23472 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20250114 08:59:29.260778 24025 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:29.261226 24025 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:29.261572 24025 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.263198 24082 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:29.113317 23972 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.216235 24167 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:29.238016 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.249964 23784 raft_consensus.cc:3049] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Stepping down as leader of term 3
I20250114 08:59:29.249791 24200 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:29.254964 24071 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.216032 23801 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.267592 21596 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:29.189824 21551 raft_consensus.cc:2949] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:29.227288 21594 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:29.268895 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1375) took 1224 ms. Trace:
I20250114 08:59:29.229243 24204 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:29.279963 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:29.289238 21703 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:29.286486 21624 rpcz_store.cc:276] 0114 08:59:28.044667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:28.044776 (+ 109us) service_pool.cc:225] Handling call
0114 08:59:28.068965 (+ 24189us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:28.069193 (+ 228us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:28.085847 (+ 16654us) log.cc:844] Serialized 14 byte log entry
0114 08:59:28.925515 (+839668us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x7b18003dc480. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:29.109829 (+184314us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b18003dc480. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c60c03a 00007fa645ac8be9 00007fa63c4cd930 00007fa63c691a14 00007fa63c691305 00007fa63f5c53d0 00007fa63f5d7666 00007fa63f5e81d7 00007fa63f5e8158
0114 08:59:29.242682 (+132853us) log.cc:844] Serialized 14 byte log entry
0114 08:59:29.268559 (+ 25877us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:29.268609 (+ 50us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:29.268656 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:29.268715 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:29.268825 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155852,"prepare.queue_time_us":16941,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":942551,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":8917,"spinlock_wait_cycles":45210624,"thread_start_us":175511,"threads_started":2,"wal-append.queue_time_us":158802}
I20250114 08:59:29.290617 21668 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:29.293601 21553 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:29.294138 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.240927 24030 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:29.301460 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:29.228513 24162 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20250114 08:59:29.300012 21521 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:29.300676 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 73474.2us
I20250114 08:59:29.271651 23935 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:29.279930 24200 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:29.295049 24226 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.872s user 0.005s sys 0.029s
I20250114 08:59:29.280580 23903 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:29.279762 23784 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:29.279788 23781 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:29.318202 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:29.301322 24191 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:29.320981 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:29.304672 24160 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:29.311041 24167 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
W20250114 08:59:29.319387 24226 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.006s sys 0.029s
I20250114 08:59:29.311753 24204 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:29.319238 24193 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250114 08:59:29.372545 24161 log.cc:926] Time spent T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250114 08:59:29.325583 24191 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:29.313164 24200 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:29.372330 24205 log.cc:926] Time spent T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250114 08:59:29.404659 22274 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:29.325837 24236 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:29.378170 24187 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.652s user 0.010s sys 0.011s
I20250114 08:59:29.321485 23781 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.370715 24162 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:29.317041 23935 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:29.310794 24030 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:29.401803 24193 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:29.429539 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 128159us
I20250114 08:59:29.402091 24167 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:29.471381 24167 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:29.424495 24187 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.704s user 0.010s sys 0.012s
I20250114 08:59:29.417263 24062 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:29.419463 24191 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:29.473433 24062 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:29.475661 24062 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:29.432212 24236 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.478605 24236 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:29.484068 23935 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:29.484851 24191 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.487154 24191 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:29.429113 24162 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:29.489441 24162 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.491125 24162 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:29.448716 23784 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:29.450031 24206 raft_consensus.cc:988] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:29.449915 24189 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:29.426364 23972 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.367484 23903 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:29.518347 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:29.518899 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 48860.8us
I20250114 08:59:29.519943 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.521029 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.431533 23899 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:29.527690 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 204660us
I20250114 08:59:29.536912 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.540441 21628 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.540395 21596 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:29.542495 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.543227 21627 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:29.563484 21669 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.564397 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.565047 21669 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:29.566543 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.567353 21623 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:29.496985 24193 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:29.573679 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.586408 21626 raft_consensus.cc:1270] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:29.579712 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.589993 21669 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.591441 21669 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:29.592192 24199 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:29.569564 24094 raft_consensus.cc:1076] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Signalling peer cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:29.589859 24226 consensus_queue.cc:1046] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:29.603770 21628 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:29.611763 21669 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.572688 24191 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:29.621088 24253 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:29.625351 23781 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.632345 21703 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:29.634006 21669 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:29.589946 24236 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:29.636936 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.588582 24252 raft_consensus.cc:2743] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:29.592952 24175 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:29.592778 24199 raft_consensus.cc:695] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:29.592442 24025 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.572888 23899 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.627111 24191 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:29.672981 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.645383 23913 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:29.645545 23472 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:29.677258 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.683070 21554 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:29.679278 21669 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.684844 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:29.687117 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.687661 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.691393 21547 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:29.697021 24243 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:29.627354 24178 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.673302 24094 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:29.707155 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:29.672555 24191 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:29.708899 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.712401 21625 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:29.681531 24188 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.644899 23874 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.691241 24175 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:29.707214 21547 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:29.733932 21548 raft_consensus.cc:2388] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:29.729578 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.724951 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.713377 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.772110 21630 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:29.773051 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.773765 21630 raft_consensus.cc:2363] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:29.729023 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.717342 21554 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:29.730090 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.774446 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.730584 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.765378 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.766831 21629 raft_consensus.cc:1235] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 08:59:29.694706 24226 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:29.697530 24243 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:29.791057 21624 raft_consensus.cc:2388] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:29.792166 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.705958 23913 raft_consensus.cc:695] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:29.800655 21550 raft_consensus.cc:2388] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:29.763243 23472 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250114 08:59:29.789494 24190 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 2.022s user 0.010s sys 0.001s
I20250114 08:59:29.722347 24178 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.704082 24186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:51560 (request call id 1) took 3082 ms. Trace:
I20250114 08:59:29.799516 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.806012 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:29.831032 21630 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:29.815539 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.816387 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.817308 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:29.797045 24243 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:29.830255 21625 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:29.763824 24094 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.846575 24030 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:29.846716 24166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57308 (request call id 476) took 5927 ms. Trace:
I20250114 08:59:29.846980 23903 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:29.829243 24199 consensus_queue.cc:237] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.847335 24166 rpcz_store.cc:276] 0114 08:59:23.913578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.633450 (+2719872us) service_pool.cc:225] Handling call
0114 08:59:26.749511 (+116061us) log.cc:844] Serialized 508 byte log entry
0114 08:59:27.869353 (+1119842us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b5400103c20. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f55ecaf 00007fa63f5477a5 00007fa63f54c9d1 00007fa63f54c959 00007fa63f54c8e9 00007fa63f54c8b1 00007fa63f54ba7d 00007fa6456d1234 00007fa6456d1069 00007fa63c694866
0114 08:59:29.840944 (+1971591us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28052,"raft.queue_time_us":244848,"raft.run_cpu_time_us":9260,"raft.run_wall_time_us":485715,"spinlock_wait_cycles":77832960,"thread_start_us":620856,"threads_started":2,"wal-append.queue_time_us":562064,"wal-append.run_cpu_time_us":8704,"wal-append.run_wall_time_us":1396135}
I20250114 08:59:29.850890 21627 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
W20250114 08:59:29.835942 24123 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.652s user 0.012s sys 0.000s
W20250114 08:59:29.840256 24082 consensus_meta.cc:287] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.577s user 0.011s sys 0.000s
I20250114 08:59:29.837046 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.857736 21554 raft_consensus.cc:2363] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:29.861419 21629 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:29.863804 21628 raft_consensus.cc:1235] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:29.866878 21594 leader_election.cc:400] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:29.867439 21594 leader_election.cc:403] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20250114 08:59:29.868265 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 229342us
I20250114 08:59:29.807499 24204 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
W20250114 08:59:29.846812 24258 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 08:59:29.892664 21624 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:29.896623 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1790) took 2567 ms. Trace:
I20250114 08:59:29.899369 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:29.905426 21624 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20250114 08:59:29.879441 24264 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:29.899231 21704 rpcz_store.cc:276] 0114 08:59:27.329342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:27.329491 (+ 149us) service_pool.cc:225] Handling call
0114 08:59:27.329876 (+ 385us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:27.476644 (+146768us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:27.476679 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:27.502514 (+ 25835us) log.cc:844] Serialized 26 byte log entry
0114 08:59:27.605512 (+102998us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:27.605551 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:27.605625 (+ 74us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:28.929247 (+1323622us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:29.695520 (+766273us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b2400245bb8. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c4eb9d4 00007fa63c4e9adb 00007fa63c578289 00007fa63c577f15 00007fa63c57755d 00007fa63f233d3e 00007fa63f24b4cc 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1
0114 08:59:29.896362 (+200842us) raft_consensus.cc:1680] finished
0114 08:59:29.896402 (+ 40us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:29.896553 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86689,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":11884,"spinlock_wait_cycles":50225920,"thread_start_us":188916,"threads_started":2,"wal-append.queue_time_us":182747}
I20250114 08:59:29.894495 24167 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:29.891914 24025 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:29.913640 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:29.864704 24082 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.903610 24253 raft_consensus.cc:695] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:29.904456 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:29.949085 21625 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250114 08:59:29.989154 21594 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:29.994398 21629 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
W20250114 08:59:29.959447 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 117374us
W20250114 08:59:29.916106 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 201939us
I20250114 08:59:30.022986 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:29.915400 24071 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.626s user 0.011s sys 0.000s
I20250114 08:59:29.943292 24175 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:29.965795 24188 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:29.949743 24200 consensus_meta.cc:287] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.522s user 0.005s sys 0.004s
W20250114 08:59:29.970997 24075 consensus_meta.cc:287] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.715s user 0.010s sys 0.002s
I20250114 08:59:30.024509 21624 raft_consensus.cc:1235] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
W20250114 08:59:29.977201 24262 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250114 08:59:29.909040 23899 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.029227 21626 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:29.919195 24187 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:29.920830 24226 raft_consensus.cc:695] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:30.032927 21553 raft_consensus.cc:1270] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:29.964789 23874 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.033753 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:29.972254 24181 raft_consensus.cc:3049] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:30.033882 21669 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:30.037009 24166 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.068809 24082 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.033051 24174 consensus_queue.cc:1035] T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20250114 08:59:30.069437 24272 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
W20250114 08:59:30.046025 24260 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.316s user 0.002s sys 0.000s
I20250114 08:59:30.070253 24272 raft_consensus.cc:695] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:30.047607 24162 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.556s user 0.010s sys 0.005s
W20250114 08:59:30.047964 24267 log.cc:926] Time spent T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20250114 08:59:30.037092 23935 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:29.829809 24186 rpcz_store.cc:276] 0114 08:59:26.603476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:26.604573 (+ 1097us) service_pool.cc:225] Handling call
0114 08:59:26.658456 (+ 53883us) log.cc:844] Serialized 508 byte log entry
0114 08:59:27.371569 (+713113us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b1800456340. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c694914 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:29.685516 (+2313947us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38,"raft.queue_time_us":19088,"raft.run_cpu_time_us":1287,"raft.run_wall_time_us":4518,"spinlock_wait_cycles":12448512,"thread_start_us":305318,"threads_started":1,"wal-append.queue_time_us":447768,"wal-append.run_cpu_time_us":2527,"wal-append.run_wall_time_us":1117100}
I20250114 08:59:30.072870 23935 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:30.075817 23935 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:30.077414 21495 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ada199404e61479593fa1b64c574bcb8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 08:59:30.026677 24273 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:30.022051 24129 raft_consensus.cc:2758] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250114 08:59:30.050868 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:30.058231 24181 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:30.065980 21629 raft_consensus.cc:1235] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 08:59:30.066505 24253 consensus_queue.cc:237] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.068333 24166 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.081851 21669 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:30.091475 21521 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:30.085759 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:30.092800 21699 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:29.983093 24094 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.080871 21622 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:30.071573 24272 consensus_queue.cc:237] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.099047 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:30.109164 21704 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:30.096873 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 224496us
I20250114 08:59:30.099251 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:30.110810 21668 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:30.112021 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:30.113562 21669 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:30.100275 24094 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:29.918655 24264 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.032516 24200 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.116225 24236 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:30.120702 24200 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.101555 24271 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:29.920018 24193 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.034121 24075 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:29.919586 23913 consensus_queue.cc:237] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.098134 21700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:46398
I20250114 08:59:30.087143 24268 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:30.143579 24175 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.144217 24268 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:30.146381 24268 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:30.147455 24266 log.cc:926] Time spent T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20250114 08:59:30.149884 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:30.150765 21622 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:30.117220 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:30.102816 24174 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:30.152639 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:30.153241 21623 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:30.154527 24261 log.cc:926] Time spent T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.425s user 0.000s sys 0.002s
I20250114 08:59:30.048739 24243 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.121918 21630 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250114 08:59:30.124963 21629 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:30.118193 24273 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:30.125522 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:30.079221 24187 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:30.127068 24094 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:30.127271 24193 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.127102 24124 raft_consensus.cc:3049] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:30.129693 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:30.130079 24264 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.130878 21628 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:30.113809 23472 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:30.122874 24271 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:30.136024 24166 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.122346 23903 consensus_queue.cc:1035] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250114 08:59:30.137343 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:30.137914 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:30.438712 21549 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250114 08:59:30.141880 21547 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:30.142606 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:30.440768 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:30.151587 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:30.420292 21552 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:30.162292 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:30.237644 21520 leader_election.cc:336] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.568s (SENT)
I20250114 08:59:30.415138 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:30.449697 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 212111us
I20250114 08:59:30.450104 21699 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
W20250114 08:59:30.450668 21520 leader_election.cc:336] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.592s (SENT)
W20250114 08:59:30.451773 21520 leader_election.cc:336] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.594s (SENT)
I20250114 08:59:30.457049 21627 raft_consensus.cc:2363] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:30.266213 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:30.416946 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:30.140435 24075 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.467872 21697 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:30.254033 21623 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
W20250114 08:59:30.161641 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 107200us
I20250114 08:59:30.480952 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1444) took 1289 ms. Trace:
I20250114 08:59:30.481376 21550 rpcz_store.cc:276] 0114 08:59:29.191761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.191884 (+ 123us) service_pool.cc:225] Handling call
0114 08:59:30.480739 (+1288855us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b540031cc54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:30.480894 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1342079360}
I20250114 08:59:30.487785 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:30.150591 24276 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:30.445698 24271 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:30.265307 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:30.421200 24124 raft_consensus.cc:738] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:30.483633 21700 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:30.500911 21700 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:30.495966 21520 leader_election.cc:336] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.608s (SENT)
I20250114 08:59:30.446436 24274 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.421320 24184 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.502712 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:30.441905 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:30.503006 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:30.509377 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:30.503576 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 386444us
I20250114 08:59:30.472337 23472 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:30.512789 21669 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:30.253486 24174 consensus_queue.cc:1035] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.318s
I20250114 08:59:30.520457 21551 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:30.493074 24193 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.521068 22269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 397) took 1302 ms. Trace:
I20250114 08:59:30.524083 21593 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:30.520334 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1446) took 1325 ms. Trace:
I20250114 08:59:30.536648 21548 rpcz_store.cc:276] 0114 08:59:29.195306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.195438 (+ 132us) service_pool.cc:225] Handling call
0114 08:59:30.520137 (+1324699us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b5400323f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:30.520278 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1176665472}
W20250114 08:59:30.505870 24191 consensus_meta.cc:287] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.730s user 0.012s sys 0.000s
I20250114 08:59:30.528628 21697 raft_consensus.cc:2463] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:30.533370 22241 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:30.487098 24033 raft_consensus.cc:3049] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:30.462016 24166 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20250114 08:59:30.538378 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:30.542032 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:30.543569 21699 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:30.522924 24274 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.581262 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:30.581954 21625 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:30.542743 24191 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.539151 24272 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:30.583035 24272 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:30.584196 21703 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:30.583981 24272 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.584007 24191 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:30.589584 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:30.591058 23801 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.591706 23801 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.592175 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:30.594871 23801 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.415693 24226 consensus_queue.cc:237] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.522532 24184 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.596577 24187 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.487337 24183 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20250114 08:59:30.611449 21699 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:30.608726 21704 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:30.534943 24123 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:30.539250 24268 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:30.540036 24033 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:30.602957 24184 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.580204 24274 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:30.539609 24193 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.545711 24094 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:30.827944 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 382535us
W20250114 08:59:30.791142 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 305652us
I20250114 08:59:30.854005 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:30.828691 23899 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.726991 24075 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:30.723062 23801 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:30.772275 24183 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:30.536353 22269 rpcz_store.cc:276] 0114 08:59:29.218966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.225822 (+ 6856us) service_pool.cc:225] Handling call
0114 08:59:30.159402 (+933580us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:30.521002 (+361600us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63705984}
I20250114 08:59:30.906229 24184 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:30.917912 24236 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.053s sys 0.051s
I20250114 08:59:31.003944 21593 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:30.999277 24236 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.057s sys 0.051s
I20250114 08:59:30.983220 21700 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:30.772868 23935 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:30.912792 24243 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:31.013566 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 480225us
I20250114 08:59:30.686982 24124 consensus_queue.cc:260] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:31.008687 24091 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s user 0.044s sys 0.083s
I20250114 08:59:31.013484 21698 raft_consensus.cc:1235] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 6. Ops: []
I20250114 08:59:30.985671 23801 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.024876 23801 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:31.025858 24188 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:30.980130 24075 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.002832 24204 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:31.009500 24236 raft_consensus.cc:695] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:31.035225 23935 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:30.986491 24183 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:31.030428 24283 log.cc:926] Time spent T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.286s user 0.001s sys 0.000s
I20250114 08:59:31.028448 24075 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.036068 24204 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.048156 24287 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:31.048655 24287 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:31.048846 24204 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.049022 24287 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:31.044729 24167 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.128s user 0.008s sys 0.005s
I20250114 08:59:31.050143 24167 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:31.050566 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:31.050789 21628 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:31.051064 24193 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:31.045421 24263 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 1.058s user 0.007s sys 0.017s
I20250114 08:59:31.013724 24184 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.058787 24184 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:30.979723 23899 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:31.045688 23913 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.034s sys 0.079s
I20250114 08:59:31.007336 24033 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:31.062552 23913 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.924s user 0.034s sys 0.079s
W20250114 08:59:31.060496 24282 log.cc:926] Time spent T 80026dc5df8c4456a48f734e1a64b930 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.327s user 0.001s sys 0.001s
W20250114 08:59:31.070022 21668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.012s sys 0.040s
W20250114 08:59:31.070394 21668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.912s user 0.012s sys 0.040s
W20250114 08:59:31.070770 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 953547us
I20250114 08:59:31.079119 21668 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:31.043401 23935 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.084957 24272 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:31.087469 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1568) took 1294 ms. Trace:
I20250114 08:59:31.088042 21551 rpcz_store.cc:276] 0114 08:59:29.793243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.793430 (+ 187us) service_pool.cc:225] Handling call
0114 08:59:30.520271 (+726841us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:30.532867 (+ 12596us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:31.087167 (+554300us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b5400323f50. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:31.087380 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1670821248}
I20250114 08:59:31.073359 24288 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.094141 21668 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:31.084395 24033 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:31.104534 24284 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:31.103334 24167 consensus_queue.cc:1046] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20250114 08:59:31.154274 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1859) took 1068 ms. Trace:
I20250114 08:59:31.154870 21703 rpcz_store.cc:276] 0114 08:59:30.085647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:30.086008 (+ 361us) service_pool.cc:225] Handling call
0114 08:59:30.097042 (+ 11034us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:30.584608 (+487566us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:31.153958 (+569350us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b5400328054. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:31.154193 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1119889152}
I20250114 08:59:31.156490 21700 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.099057 24236 consensus_queue.cc:237] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.098858 24288 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.164337 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1583) took 1437 ms. Trace:
I20250114 08:59:31.166720 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1854) took 1116 ms. Trace:
I20250114 08:59:31.176885 21630 rpcz_store.cc:276] 0114 08:59:29.727195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.831665 (+104470us) service_pool.cc:225] Handling call
0114 08:59:29.832061 (+ 396us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:30.420169 (+588108us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:31.156416 (+736247us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54003b97d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:31.164249 (+ 7833us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665883520}
I20250114 08:59:31.177031 21698 rpcz_store.cc:276] 0114 08:59:30.049931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:30.051210 (+ 1279us) service_pool.cc:225] Handling call
0114 08:59:30.065452 (+ 14242us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:31.019100 (+953648us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:31.160842 (+141742us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54003705d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:31.166637 (+ 5795us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2179852032}
I20250114 08:59:31.168115 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:46398 (request call id 1857) took 1103 ms. Trace:
I20250114 08:59:31.178773 21700 rpcz_store.cc:276] 0114 08:59:30.084317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:30.085935 (+ 1618us) service_pool.cc:225] Handling call
0114 08:59:31.155147 (+1069212us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54003705d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5c4da7 00007fa64452ded0 00007fa63dc05794 00007fa63dc05722 00007fa63dc05651 00007fa63dc055cc 00007fa63dc04882 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:31.157057 (+ 1910us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":807107840}
I20250114 08:59:31.182847 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.183905 21630 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:31.185330 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:31.185353 21520 leader_election.cc:336] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.518s (SENT)
W20250114 08:59:31.186791 21520 leader_election.cc:336] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.582s (SENT)
I20250114 08:59:31.187269 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:31.188831 21520 leader_election.cc:336] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.855s (SENT)
W20250114 08:59:31.190047 21520 leader_election.cc:336] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.894s (SENT)
I20250114 08:59:31.190516 21520 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:31.191540 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.192173 21623 raft_consensus.cc:2371] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4. Re-sending same reply.
I20250114 08:59:31.150379 24204 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:31.200299 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.201153 21625 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:31.198101 24243 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:31.206938 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.209033 21624 raft_consensus.cc:1270] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:30.495352 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:31.119855 24181 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.213164 24204 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.218588 21703 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 08:59:31.219048 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.263967 21630 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:31.241763 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.265499 21624 raft_consensus.cc:2363] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:31.266964 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.267446 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.267773 21624 raft_consensus.cc:2371] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
W20250114 08:59:31.206132 24289 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250114 08:59:31.269260 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.296886 21630 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:31.217746 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.297720 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.298563 21628 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:31.181670 24284 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.248816 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.320483 21626 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:31.271579 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.218779 24228 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:31.250602 24243 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.251639 24183 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.260020 24280 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:31.218262 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.346532 21629 raft_consensus.cc:2371] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9. Re-sending same reply.
I20250114 08:59:31.293321 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:31.294153 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:31.198585 24208 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:31.295670 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.352542 21624 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 6
I20250114 08:59:31.248317 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.299039 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.318102 21625 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:31.198940 24193 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:31.321714 21700 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:31.332180 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.349400 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.216852 24293 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:31.360013 21550 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:31.366200 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.358713 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.247298 24204 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:31.385241 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 478712us
I20250114 08:59:31.385951 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.318650 23784 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:31.370056 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.398581 21703 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:31.359802 21627 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:31.380918 21594 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:31.317512 24226 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.012s sys 0.044s
I20250114 08:59:31.386788 21629 raft_consensus.cc:2371] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8. Re-sending same reply.
I20250114 08:59:31.386768 22240 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:31.396976 21521 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:31.416744 24030 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:31.417178 24030 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:31.417543 24030 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:31.408390 21624 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:31.419708 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.409459 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.423055 21624 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 4.
I20250114 08:59:31.417539 21521 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:31.429049 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.430469 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.431098 22241 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:31.431201 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.435493 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.433586 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.431069 21627 raft_consensus.cc:2371] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
I20250114 08:59:31.432929 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.437872 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.433095 21521 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:31.411899 24226 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s user 0.012s sys 0.044s
I20250114 08:59:31.385285 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 259) took 2134 ms. Trace:
I20250114 08:59:31.369567 23899 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.429021 24293 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.426620 24025 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:31.445029 24292 log.cc:926] Time spent T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.251s user 0.000s sys 0.000s
W20250114 08:59:31.445971 24290 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.280s user 0.000s sys 0.000s
I20250114 08:59:31.358989 24280 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.387091 23935 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.410383 24204 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:31.471199 24075 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:31.458261 21630 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:31.459403 21626 raft_consensus.cc:2371] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9. Re-sending same reply.
I20250114 08:59:31.460495 21629 raft_consensus.cc:2463] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:31.464360 21521 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:31.471063 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:31.439169 24208 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:31.494235 24193 raft_consensus.cc:2798] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:31.348011 24228 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.494925 24193 raft_consensus.cc:695] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:31.496060 24193 consensus_queue.cc:237] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.497833 24228 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:31.432762 24186 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.478394 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.498739 24186 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.499547 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.500422 21627 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:31.501407 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.501626 24186 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.502063 21627 raft_consensus.cc:2371] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
I20250114 08:59:31.448168 24030 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.505494 24030 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.420881 24187 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:31.503180 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.507074 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:31.492926 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.509662 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.510941 21627 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:31.532006 21627 raft_consensus.cc:1270] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 2. (term mismatch)
I20250114 08:59:31.515645 21521 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:31.533339 24030 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:31.534003 21630 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:31.533969 24030 raft_consensus.cc:695] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:31.535316 21521 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:31.470896 24295 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:31.362301 24284 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.545992 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.479220 24274 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:31.481406 24264 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:31.549190 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.552891 21627 pending_rounds.cc:77] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 1
I20250114 08:59:31.627115 21627 pending_rounds.cc:99] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 08:59:31.627441 21627 raft_consensus.cc:2883] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:31.565356 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.545930 24288 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.478400 24025 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.490293 24075 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:31.541682 24204 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:31.653074 24075 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:31.654464 24204 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:31.635986 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.657429 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.658533 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.659073 21629 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:31.659812 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.660082 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.636132 24030 consensus_queue.cc:237] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.660758 21630 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:31.661526 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.662164 21630 raft_consensus.cc:2363] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:31.662824 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.663338 21630 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:31.629074 24274 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.629650 24264 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:31.645942 24025 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.646464 21624 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:31.647106 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.621407 24094 raft_consensus.cc:2798] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:31.649623 24280 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:31.650763 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:31.651130 24293 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.632406 23899 consensus_queue.cc:1046] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250114 08:59:31.687057 23899 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:31.687494 23899 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:31.688448 23899 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.689478 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.689868 23899 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:31.690128 21624 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:31.701097 21630 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:31.702045 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.669700 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.672945 24243 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.683202 21626 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:31.668166 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:31.753113 21629 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:31.461374 22267 rpcz_store.cc:276] 0114 08:59:29.223438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.225991 (+ 2553us) service_pool.cc:225] Handling call
0114 08:59:30.043267 (+817276us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:30.043487 (+ 220us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:30.076904 (+ 33417us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:30.076937 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:30.076996 (+ 59us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:30.077036 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:31.300545 (+1223509us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b5400710554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:31.332862 (+ 32317us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:31.358037 (+ 25175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125651200}
W20250114 08:59:31.704913 21520 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.014s sys 0.020s
W20250114 08:59:31.754338 21520 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.560s user 0.014s sys 0.020s
W20250114 08:59:31.754757 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 560575us
W20250114 08:59:31.705348 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 494741us
I20250114 08:59:31.709147 21624 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:31.716948 24284 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.699353 24264 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
W20250114 08:59:31.717456 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 274397us
I20250114 08:59:31.633747 24091 consensus_queue.cc:1046] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20250114 08:59:31.753758 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.756426 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.752542 21627 raft_consensus.cc:2371] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8. Re-sending same reply.
W20250114 08:59:31.771733 21520 leader_election.cc:336] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.758s (SENT)
I20250114 08:59:31.793938 21520 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:31.794023 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.795109 21630 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:31.795228 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.778079 21624 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:31.797188 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.797765 21624 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:31.798009 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.798583 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.799369 21624 raft_consensus.cc:2405] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:31.804649 21629 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:31.794960 21626 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:31.795864 21703 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:31.722375 24293 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:31.816254 21624 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250114 08:59:31.816037 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 44359.3us
I20250114 08:59:31.825745 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.818727 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:31.827818 21551 raft_consensus.cc:2405] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:31.822817 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.829015 21627 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:31.771517 24280 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:31.823336 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.777634 24129 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:31.851467 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 231237us
I20250114 08:59:31.853884 24094 raft_consensus.cc:695] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:31.855100 24094 consensus_queue.cc:237] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:31.857193 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 155540us
I20250114 08:59:31.859016 22240 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:31.859898 24295 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:31.860273 24295 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.860666 24295 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:31.868089 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.869110 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.868335 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.869932 21630 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:31.870272 21629 raft_consensus.cc:2388] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:31.871682 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:31.872438 21550 raft_consensus.cc:2388] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:31.874073 21547 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:31.874670 21547 raft_consensus.cc:604] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:31.875070 21547 raft_consensus.cc:612] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:31.794607 22269 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:31.794984 24208 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:31.808854 24191 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.878320 24091 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:31.916445 21439 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b3650a91ff4a4213b9091f197bf75fb9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250114 08:59:31.916386 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:31.917603 21439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 140098us
I20250114 08:59:31.885337 24193 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:31.926824 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.929790 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.881557 24129 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.933682 24274 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.934494 24274 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.934957 24274 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.935324 24129 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:31.936762 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.937355 24311 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:31.937919 24311 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.940076 24311 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.902572 24188 raft_consensus.cc:3049] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:31.940665 24188 raft_consensus.cc:738] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:31.941598 24188 consensus_queue.cc:260] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.942672 24188 raft_consensus.cc:3054] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:31.892735 24191 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.945433 24091 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:31.946316 24191 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.873020 24280 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.950073 24280 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:31.921898 24175 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:31.951031 24175 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.952561 24314 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:31.953133 24314 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.955170 24314 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.955462 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:31.955705 24193 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:31.957875 24193 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.836444 24293 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:31.959733 21700 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:31.960289 21548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:31.960836 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:31.961138 21548 raft_consensus.cc:604] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:31.917927 24271 consensus_meta.cc:287] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.387s user 0.011s sys 0.000s
I20250114 08:59:31.961783 24271 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.963841 24271 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:31.889755 24208 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:31.966452 24208 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:31.967051 21547 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:31.969869 22240 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:31.937536 21698 raft_consensus.cc:2363] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5.
I20250114 08:59:31.909276 24124 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:31.978603 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:31.979683 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 126165us
I20250114 08:59:31.981060 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.981853 21704 raft_consensus.cc:2388] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:31.981763 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.982728 21695 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:31.983767 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:31.984809 21696 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:31.986979 21704 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:31.995471 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:31.909024 24243 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:31.953693 24287 consensus_meta.cc:287] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.904s user 0.008s sys 0.002s
I20250114 08:59:32.123795 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.006390 21521 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:32.126441 21554 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250114 08:59:32.041831 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.043910 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:32.209715 21552 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:32.208696 21554 pending_rounds.cc:77] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting all ops after (but not including) 3
I20250114 08:59:32.210817 21554 pending_rounds.cc:99] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250114 08:59:32.123718 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.211732 21554 raft_consensus.cc:2883] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:32.146337 21594 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:32.211222 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 204900us
I20250114 08:59:32.213184 21521 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:32.042579 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.040848 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.005374 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:31.995926 23784 consensus_meta.cc:287] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.585s user 0.009s sys 0.001s
I20250114 08:59:32.197410 24274 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:32.225069 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 78835.8us
I20250114 08:59:32.220000 24322 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:32.178037 23903 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.239413 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.281855 24322 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:32.214246 21451 catalog_manager.cc:5526] T 82452b55145444529bf0609cd7767adf P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 7 to 10, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132). New cstate: current_term: 10 leader_uuid: "57f9cc1dd3c64647848a33a65b08c893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:32.286561 21702 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:31.977694 24307 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:32.288360 21697 raft_consensus.cc:2363] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:32.222673 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.234797 21552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:32.291688 21552 raft_consensus.cc:604] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:32.042160 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.282336 21554 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 10.
I20250114 08:59:32.299137 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.004812 24030 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:32.202306 24316 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.209161 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:32.308156 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.282316 24322 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:32.308483 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.311009 21704 raft_consensus.cc:1270] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:31.976173 24271 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.225170 24189 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:32.127049 24093 raft_consensus.cc:3049] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:32.287990 21621 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:32.320453 21546 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 08:59:32.282927 24166 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:32.282096 23903 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.061156 24293 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:32.060290 24075 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.056757 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:32.062397 23913 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.352535 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.353792 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.368947 21554 raft_consensus.cc:2388] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:32.372478 21668 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:32.341873 24271 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.377205 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.377661 24271 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.378610 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.379807 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.380801 21548 raft_consensus.cc:1235] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250114 08:59:32.380931 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.381400 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.381979 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.382694 21701 raft_consensus.cc:2363] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:32.382551 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.383193 21704 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:32.383440 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.384004 21701 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:32.335732 24189 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:32.384720 24189 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:32.385030 21695 raft_consensus.cc:1235] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:32.385084 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.386266 21695 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:32.386504 21702 raft_consensus.cc:1235] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:32.387236 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:32.388144 21696 raft_consensus.cc:2388] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:32.297129 24287 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.341027 24166 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:32.390497 24287 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.390657 24166 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.335772 24268 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.335503 24093 raft_consensus.cc:738] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:32.393396 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1416) took 1178 ms. Trace:
I20250114 08:59:32.393155 24268 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.394032 24325 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:32.393738 24093 consensus_queue.cc:260] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.394506 24325 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:32.394817 24093 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 6
I20250114 08:59:32.394907 24325 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.387193 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.381083 23913 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.370054 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.221269 24243 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.393779 21623 rpcz_store.cc:276] 0114 08:59:31.214873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:31.410907 (+196034us) service_pool.cc:225] Handling call
0114 08:59:31.433041 (+ 22134us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:31.433352 (+ 311us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:31.433382 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:31.442898 (+ 9516us) log.cc:844] Serialized 26 byte log entry
0114 08:59:31.786152 (+343254us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:31.786188 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:31.786253 (+ 65us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:32.393180 (+606927us) raft_consensus.cc:1680] finished
0114 08:59:32.393199 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:32.393338 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":221694,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":17280,"thread_start_us":342886,"threads_started":1,"wal-append.queue_time_us":758095}
I20250114 08:59:32.400303 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:32.406739 21626 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:32.407338 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 84634.3us
I20250114 08:59:32.407814 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:32.059715 24295 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.417299 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1415) took 1211 ms. Trace:
I20250114 08:59:32.417613 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.397156 24271 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:32.408594 21630 raft_consensus.cc:1270] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:32.420001 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.417730 21625 rpcz_store.cc:276] 0114 08:59:31.205647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:31.354438 (+148791us) service_pool.cc:225] Handling call
0114 08:59:31.354868 (+ 430us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:31.361629 (+ 6761us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:31.361657 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:31.387699 (+ 26042us) log.cc:844] Serialized 26 byte log entry
0114 08:59:31.881733 (+494034us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:31.976072 (+ 94339us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b1800439840. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c691305 00007fa642fa2107 00007fa63f5d78f7 00007fa63f5e700f 00007fa63f5e6f88
0114 08:59:32.407278 (+431206us) log.cc:844] Serialized 14 byte log entry
0114 08:59:32.407558 (+ 280us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:32.407646 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:32.407700 (+ 54us) raft_consensus.cc:1680] finished
0114 08:59:32.417074 (+ 9374us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:32.417232 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19550,"raft.queue_time_us":7362,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":31582,"spinlock_wait_cycles":13863168,"thread_start_us":853239,"threads_started":2,"wal-append.queue_time_us":490771}
W20250114 08:59:32.404228 21520 leader_election.cc:336] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.034s (SENT)
I20250114 08:59:32.423823 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:32.425398 21520 leader_election.cc:336] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.068s (SENT)
I20250114 08:59:32.425801 21520 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:32.427373 21520 leader_election.cc:336] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.201s (SENT)
I20250114 08:59:32.427738 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:32.429184 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:32.430245 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.319303 24030 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.432874 21704 raft_consensus.cc:2405] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:32.435046 21626 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:32.440799 22240 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:32.439546 21520 leader_election.cc:336] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.313s (SENT)
I20250114 08:59:32.443495 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.445531 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:32.447824 21704 raft_consensus.cc:2405] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250114 08:59:32.453274 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.456212 21554 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:32.463166 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.326821 24316 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.429081 24193 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:32.430737 23801 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250114 08:59:32.450112 24033 consensus_meta.cc:287] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 1.338s user 0.012s sys 0.001s
I20250114 08:59:32.540297 21554 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:32.441929 24324 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:32.545989 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.546852 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.557353 21699 raft_consensus.cc:2363] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
W20250114 08:59:32.533490 24313 log.cc:926] Time spent T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250114 08:59:32.561698 21552 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 08:59:32.562547 21699 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250114 08:59:32.563519 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.563911 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:32.564265 21703 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
W20250114 08:59:32.555717 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 101787us
I20250114 08:59:32.559195 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.559465 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.565850 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.565940 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.566568 21695 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:32.567214 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.567631 21695 raft_consensus.cc:1270] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 08:59:32.450309 24279 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:32.569303 24268 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:32.575726 21669 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:32.577930 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.590651 21548 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:32.598464 21548 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:32.594419 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:32.601969 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:32.609035 21549 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:32.610080 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.618634 21669 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:32.547406 24326 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:32.593919 24075 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:32.594077 24330 consensus_queue.cc:1035] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.031s
I20250114 08:59:32.625490 24330 consensus_queue.cc:1225] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:32.625736 23784 consensus_queue.cc:1046] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:32.626585 23784 raft_consensus.cc:3049] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:32.626909 23784 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:32.586140 24337 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.627652 24337 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.628886 24330 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:32.629833 24337 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:32.629448 24330 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.592818 24033 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:32.630609 24033 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:32.631043 24033 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.632530 24330 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:32.634718 21621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1401) took 3115 ms (client timeout 1669 ms). Trace:
W20250114 08:59:32.635275 21621 rpcz_store.cc:269] 0114 08:59:29.519513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:29.519663 (+ 150us) service_pool.cc:225] Handling call
0114 08:59:32.634460 (+3114797us) spinlock_profiling.cc:243] Waited 2.75 s on lock 0x7b540022cc54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:32.634620 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6319144704}
I20250114 08:59:32.594261 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:32.637050 22269 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 4.
I20250114 08:59:32.592164 23899 consensus_queue.cc:1046] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20250114 08:59:32.638304 21593 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:32.638985 24243 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:32.639201 24071 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:32.639653 24071 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:32.586419 24181 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:32.640098 24071 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:32.630097 21623 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:55318
I20250114 08:59:32.570272 24284 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.641041 21623 raft_consensus.cc:604] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [term 3 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:32.641652 21552 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10. Ops: []
I20250114 08:59:32.642635 21548 raft_consensus.cc:1235] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:32.578897 24271 raft_consensus.cc:2758] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20250114 08:59:32.650482 21548 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:32.593091 23913 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.587991 24316 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:32.577780 24279 raft_consensus.cc:695] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:32.621551 24199 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250114 08:59:32.669166 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 103251us
I20250114 08:59:32.676199 21701 raft_consensus.cc:2405] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:32.650682 24337 raft_consensus.cc:3049] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Stepping down as leader of term 7
W20250114 08:59:32.664777 24264 consensus_meta.cc:287] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.847s user 0.007s sys 0.004s
I20250114 08:59:32.684842 24075 consensus_queue.cc:1046] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:32.686089 24343 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20250114 08:59:32.687506 24333 log.cc:926] Time spent T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20250114 08:59:32.688113 24295 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.689250 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:32.646405 24189 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.695264 21623 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.708781 24030 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:32.710121 24075 raft_consensus.cc:3049] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:32.661060 24268 consensus_queue.cc:1046] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20250114 08:59:32.654812 24284 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.645603 24326 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.673604 24253 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20250114 08:59:32.627666 23784 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.783092 21555 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:32.801970 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 109990us
I20250114 08:59:32.721338 22270 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:32.765057 21700 raft_consensus.cc:1270] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250114 08:59:32.790555 21451 catalog_manager.cc:5526] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 7 to 8, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 8 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:32.682823 24274 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:32.681816 24337 raft_consensus.cc:738] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:32.819429 21451 catalog_manager.cc:5526] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:32.814364 21622 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:32.681130 23903 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.677359 24272 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:32.817904 21550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:32.823408 21520 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:32.771876 24320 log.cc:926] Time spent T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.321s user 0.000s sys 0.001s
W20250114 08:59:32.713569 24124 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.727s user 0.010s sys 0.000s
I20250114 08:59:32.826472 21451 catalog_manager.cc:5526] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 4, leader changed from 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:59:32.725214 24188 consensus_meta.cc:287] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.782s user 0.010s sys 0.003s
I20250114 08:59:32.778829 24316 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.832437 21699 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:32.834141 21549 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:32.830087 21550 raft_consensus.cc:604] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:32.775729 24025 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.734220 24295 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.767799 24175 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.800689 24075 raft_consensus.cc:738] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:32.857120 24271 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.871124 24271 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.871218 24075 consensus_queue.cc:260] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.872485 24075 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.683048 24166 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.873291 24271 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.874923 24166 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:32.740810 24243 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.875465 24071 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.852367 24336 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:32.875979 24243 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.876802 24336 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:32.877223 24336 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:32.878527 24071 leader_election.cc:290] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:32.878711 24243 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.879778 21629 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:32.851186 21622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43436 (request call id 267) took 2662 ms (client timeout 1925 ms). Trace:
I20250114 08:59:32.882366 24326 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:32.882500 21622 rpcz_store.cc:269] 0114 08:59:30.189000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:30.189215 (+ 215us) service_pool.cc:225] Handling call
0114 08:59:32.850992 (+2661777us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54001522d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:32.851110 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5438531840}
I20250114 08:59:32.855134 21520 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:32.828052 24268 consensus_queue.cc:1225] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250114 08:59:32.825232 24297 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.848378 21623 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:32.695405 23801 consensus_queue.cc:1046] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.156s
I20250114 08:59:32.829919 24274 raft_consensus.cc:695] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:32.696199 24327 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.161s
I20250114 08:59:32.849824 24345 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20250114 08:59:32.847272 24199 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:32.905131 24345 raft_consensus.cc:3049] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:32.905202 24124 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:32.838415 24030 consensus_queue.cc:1046] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20250114 08:59:32.883698 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:32.851970 21550 raft_consensus.cc:612] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:32.884747 21520 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:32.910604 22270 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:32.711807 24325 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.722479 24322 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.895787 22240 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:32.898979 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1420) took 1640 ms. Trace:
I20250114 08:59:32.899387 24183 raft_consensus.cc:3049] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:32.886641 24319 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:32.864181 24025 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.905866 24124 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:32.902019 24033 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.903985 23784 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:32.847193 24094 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.909070 21451 catalog_manager.cc:5526] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:32.691481 24264 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.913939 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:32.914989 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:59:32.915828 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:32.946441 21520 leader_election.cc:400] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 7. Current term is 8.
I20250114 08:59:32.947134 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:32.947803 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:32.915901 21628 rpcz_store.cc:276] 0114 08:59:31.258672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:31.438572 (+179900us) service_pool.cc:225] Handling call
0114 08:59:31.448851 (+ 10279us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:31.655602 (+206751us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:31.655631 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:31.656101 (+ 470us) log.cc:844] Serialized 26 byte log entry
0114 08:59:31.860856 (+204755us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:32.259313 (+398457us) log.cc:844] Serialized 14 byte log entry
0114 08:59:32.281286 (+ 21973us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:32.453531 (+172245us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002909d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:32.453589 (+ 58us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:32.588223 (+134634us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800196960. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:32.771145 (+182922us) spinlock_profiling.cc:243] Waited 9.93 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:32.898664 (+127519us) raft_consensus.cc:1680] finished
0114 08:59:32.898696 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:32.898901 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":427398,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":29493,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":17376,"spinlock_wait_cycles":522724224,"thread_start_us":526661,"threads_started":2,"wal-append.queue_time_us":741390}
I20250114 08:59:32.774595 24279 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.886313 24284 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:32.945116 21451 catalog_manager.cc:5526] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
W20250114 08:59:32.925725 21453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55874 (request call id 199) took 16820 ms (client timeout 15000 ms). Trace:
I20250114 08:59:32.905375 24199 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.892575 24337 consensus_queue.cc:260] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.860442 24316 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:32.919538 24183 raft_consensus.cc:738] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:32.967358 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.883663 24295 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:32.905762 24345 raft_consensus.cc:738] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:32.939393 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.948683 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.978363 21555 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.941612 24334 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:32.921336 23801 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:32.949255 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:32.981729 23801 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:32.982266 23801 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:32.949647 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:32.953075 24319 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:32.985435 24319 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:32.985956 24319 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:32.993969 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.996935 21696 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:32.995697 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:32.998869 21704 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:32.963675 22270 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:32.907119 24297 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.023403 24297 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:32.941375 24291 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:32.968372 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.973249 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.024706 21698 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:32.975040 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:32.976083 21695 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20250114 08:59:32.954141 21453 rpcz_store.cc:269] 0114 08:59:16.210476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:16.219869 (+ 9393us) service_pool.cc:225] Handling call
0114 08:59:32.921353 (+16701484us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:28.285260 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:28.478111 (+192851us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:28.478172 (+ 61us) write_op.cc:435] Acquired schema lock
0114 08:59:28.478185 (+ 13us) tablet.cc:659] Decoding operations
0114 08:59:28.514715 (+ 36530us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:28.514842 (+ 127us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:28.514892 (+ 50us) tablet.cc:682] Acquiring locks for 29 operations
0114 08:59:28.523787 (+ 8895us) tablet.cc:698] Row locks acquired
0114 08:59:28.523821 (+ 34us) write_op.cc:260] PREPARE: finished
0114 08:59:28.524049 (+ 228us) write_op.cc:270] Start()
0114 08:59:28.524193 (+ 144us) write_op.cc:276] Timestamp: P: 1736845168523974 usec, L: 0
0114 08:59:28.524222 (+ 29us) op_driver.cc:348] REPLICATION: starting
0114 08:59:28.750261 (+226039us) log.cc:844] Serialized 11484 byte log entry
0114 08:59:29.620913 (+870652us) op_driver.cc:464] REPLICATION: finished
0114 08:59:29.896764 (+275851us) write_op.cc:301] APPLY: starting
0114 08:59:29.909186 (+ 12422us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:30.025617 (+116431us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:30.025748 (+ 131us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:30.065295 (+ 39547us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:30.065990 (+ 695us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=29
0114 08:59:30.066061 (+ 71us) write_op.cc:312] APPLY: finished
0114 08:59:30.066873 (+ 812us) log.cc:844] Serialized 193 byte log entry
0114 08:59:30.078916 (+ 12043us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:30.097663 (+ 18747us) write_op.cc:454] Released schema lock
0114 08:59:30.101127 (+ 3464us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:30.101603 (+ 476us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:30.101631 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26372,"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":220234,"apply.run_cpu_time_us":8727,"apply.run_wall_time_us":246560,"num_ops":29,"prepare.queue_time_us":162052,"prepare.run_cpu_time_us":43352,"prepare.run_wall_time_us":793714,"raft.queue_time_us":388661,"raft.run_cpu_time_us":13649,"raft.run_wall_time_us":120676,"replication_time_us":1091255,"thread_start_us":750679,"threads_started":4,"wal-append.queue_time_us":418454,"wal-append.run_cpu_time_us":58547,"wal-append.run_wall_time_us":1900790}]]}
I20250114 08:59:33.041038 21546 raft_consensus.cc:1235] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:33.042515 21593 leader_election.cc:304] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.043486 21703 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 08:59:33.043651 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:33.044625 22270 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:33.044835 21700 raft_consensus.cc:1270] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:33.045975 21593 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.042812 21555 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
W20250114 08:59:33.047833 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 74979.6us
I20250114 08:59:33.046191 21698 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:33.038239 24334 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:33.050560 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 431982us
I20250114 08:59:32.968225 24268 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:32.966385 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:33.036593 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:33.035382 24310 raft_consensus.cc:3049] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:33.037045 24181 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:33.055243 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.062630 21669 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:32.949419 24094 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:33.063851 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 114660us
I20250114 08:59:33.064812 21520 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.047585 24297 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election won for term 5
I20250114 08:59:33.049392 24337 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:33.050237 24291 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:33.067873 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.041702 24189 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.069780 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.081703 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.097198 24025 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.059926 24268 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.101136 24268 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.102985 24295 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.047130 24345 consensus_queue.cc:260] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.104730 24345 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:33.110743 24199 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:33.111325 24033 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:33.113780 24322 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.114244 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.114940 21551 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:33.047394 24274 consensus_queue.cc:237] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.039168 24253 consensus_queue.cc:1046] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20250114 08:59:33.118733 22267 raft_consensus.cc:2388] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:33.057571 24336 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.050079 24183 consensus_queue.cc:260] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.122256 24094 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:33.064507 24272 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:33.062757 24354 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:33.068363 24352 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20250114 08:59:33.068732 24188 consensus_queue.cc:1046] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20250114 08:59:33.088852 24343 raft_consensus.cc:2798] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:33.133404 24316 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.070366 24344 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:33.113809 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.165421 21555 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:33.125739 24175 raft_consensus.cc:2743] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:33.139451 21594 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.130914 24346 raft_consensus.cc:3049] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:33.174036 21594 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.131525 24324 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250114 08:59:33.062307 24310 raft_consensus.cc:738] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:33.065376 24264 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.154883 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.196066 21553 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:33.147111 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:33.157480 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 87756.5us
I20250114 08:59:33.151676 24025 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:33.067670 24326 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:33.159817 21521 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:33.059548 24325 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:33.038635 24124 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.181560 24268 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:33.141811 24253 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:33.137778 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 44657.6us
I20250114 08:59:33.161211 24353 consensus_queue.cc:1046] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250114 08:59:33.063871 24181 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.159191 24184 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:33.160734 24354 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:33.140817 21546 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
W20250114 08:59:33.186087 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 58776us
I20250114 08:59:33.197333 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.198027 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.173154 24343 raft_consensus.cc:695] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:33.173338 24346 raft_consensus.cc:738] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:33.236313 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.182138 24316 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:33.268000 24316 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:33.268311 21627 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:33.268415 24316 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:33.268679 24336 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.232686 24184 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:33.270243 24184 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:33.270915 21623 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:33.271965 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.125691 24334 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.264657 24354 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:33.273933 21623 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:33.275223 21623 raft_consensus.cc:1270] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 08:59:33.276408 21623 raft_consensus.cc:1270] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:33.277999 24343 consensus_queue.cc:237] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.237571 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.247079 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:33.182299 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:33.126380 24291 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.177139 24183 raft_consensus.cc:3054] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:33.235797 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.299079 21629 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:33.299903 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:33.266515 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 68561.9us
I20250114 08:59:33.269634 24346 consensus_queue.cc:260] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.207861 21521 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:33.250898 21624 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:33.140333 24297 raft_consensus.cc:695] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:33.286564 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.304036 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.291714 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.292861 24033 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:33.265967 24357 log.cc:926] Time spent T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20250114 08:59:33.309984 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.296198 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.310683 21629 pending_rounds.cc:77] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 1
I20250114 08:59:33.310973 21629 pending_rounds.cc:99] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 5.2
I20250114 08:59:33.293300 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.311192 21629 raft_consensus.cc:2883] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:33.301718 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.300549 21701 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:33.311872 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.305789 21697 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:33.281657 24094 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:33.307664 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.312629 21704 raft_consensus.cc:2388] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
W20250114 08:59:33.312989 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 105212us
I20250114 08:59:33.301960 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.207957 24351 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:33.312418 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.315302 21701 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
W20250114 08:59:33.315529 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 151174us
I20250114 08:59:33.313391 21624 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:33.209161 24268 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:33.280112 24264 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:33.319164 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.323964 21699 raft_consensus.cc:1235] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 6. Ops: []
I20250114 08:59:33.318564 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.326567 21520 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.328408 21596 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.301213 24284 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:33.232455 24075 consensus_queue.cc:1035] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20250114 08:59:33.216843 24327 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
I20250114 08:59:33.326632 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.347855 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.331684 21624 raft_consensus.cc:1270] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:33.234175 24208 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20250114 08:59:33.334743 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.350040 21698 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:33.199620 24319 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.320195 24094 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:33.345506 22241 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:33.342020 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:33.353096 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.351627 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.348762 21626 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:33.232235 24253 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.335943 24264 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:33.327185 24268 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:33.388999 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.320811 24351 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:33.389896 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.391562 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:33.329407 24167 raft_consensus.cc:3049] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:33.328820 24093 raft_consensus.cc:2798] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:33.288192 24310 consensus_queue.cc:260] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.398447 22241 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:33.206149 24025 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.386796 21623 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:33.313087 24033 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.410336 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:33.465085 21669 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.541654 21624 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:33.541833 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 76867.1us
I20250114 08:59:33.542162 21628 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:33.308718 22273 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:33.340143 24193 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election won for term 10
W20250114 08:59:33.425747 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 83809.3us
I20250114 08:59:33.540668 21625 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:33.550962 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
W20250114 08:59:33.540380 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 142043us
I20250114 08:59:33.335577 24189 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250114 08:59:33.468694 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 138580us
I20250114 08:59:33.562572 21622 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 08:59:33.566241 22239 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.352344 24346 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:33.568897 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:33.574402 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 225040us
I20250114 08:59:33.575135 21623 raft_consensus.cc:1235] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10. Ops: [6.2-9.3]
I20250114 08:59:33.353439 24071 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:33.572800 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:33.551051 24344 raft_consensus.cc:988] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:33.570637 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.631983 21623 raft_consensus.cc:1235] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 4. Current term is 5. Ops: []
I20250114 08:59:33.529095 24353 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:33.543608 24298 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:33.549690 24319 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:33.559103 24264 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:33.397514 24094 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:33.431543 24166 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:33.639668 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.562191 24181 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:33.542614 24167 raft_consensus.cc:738] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:33.577204 24315 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:33.439994 23801 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.426100 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:33.457315 24297 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.669528 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.560714 24351 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:33.674069 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.562469 24189 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:33.677489 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 347314us
I20250114 08:59:33.565482 24352 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:33.436012 24326 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:33.471446 24175 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.166s
I20250114 08:59:33.572881 24287 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:33.637555 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:33.700254 21553 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:33.764936 21700 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:33.708956 21555 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:33.569317 24093 raft_consensus.cc:695] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:33.581152 24193 raft_consensus.cc:695] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:33.565090 24342 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:33.456874 24336 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.220s
I20250114 08:59:33.529307 24184 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.473278 24272 consensus_queue.cc:1046] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.205s
I20250114 08:59:33.350395 24284 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.663161 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.525290 24354 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:33.449281 24316 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.573104 24325 consensus_queue.cc:1035] T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.139s
I20250114 08:59:33.576712 24356 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.142s
I20250114 08:59:33.738149 21546 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 9.
I20250114 08:59:33.677198 24298 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:33.745123 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.697530 24166 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:33.736747 23784 consensus_meta.cc:287] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.787s user 0.011s sys 0.000s
I20250114 08:59:33.749840 21547 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:33.750875 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
W20250114 08:59:33.772823 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 202250us
W20250114 08:59:33.772534 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 211739us
I20250114 08:59:33.849474 21630 raft_consensus.cc:2363] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:33.849607 21596 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:33.809628 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:33.812177 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 110829us
I20250114 08:59:33.818079 21698 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:33.850529 21669 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:33.851102 21549 raft_consensus.cc:2388] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:33.745961 24378 raft_consensus.cc:1076] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Signalling peer cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:33.736526 24315 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.747320 24352 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:33.803900 24243 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250114 08:59:33.862882 24364 log.cc:926] Time spent T 00000000000000000000000000000000 P 221e58f7b595419f9fb78c5c0e6207b5: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
I20250114 08:59:33.863523 23784 raft_consensus.cc:2743] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:33.865044 24352 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:33.866591 24183 consensus_meta.cc:287] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.568s user 0.006s sys 0.003s
I20250114 08:59:33.813359 24167 consensus_queue.cc:260] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.867724 24167 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:33.842515 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 98760.7us
I20250114 08:59:33.873701 21520 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.875200 21520 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.749477 24253 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:33.839301 24361 log.cc:926] Time spent T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.398s user 0.000s sys 0.001s
W20250114 08:59:33.848510 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 97713.4us
I20250114 08:59:33.847015 24166 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:33.834444 24175 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:33.858476 21596 leader_election.cc:400] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5.
I20250114 08:59:33.745855 24189 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:33.844326 24356 consensus_queue.cc:1225] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer d7732324fb5f48d3b16a29d4bb2584ed log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 08:59:33.856195 24093 consensus_queue.cc:237] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.847195 24298 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:33.858317 24272 raft_consensus.cc:3049] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:33.861639 24193 consensus_queue.cc:237] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.3, Last appended by leader: 3, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.862538 24184 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.685462 24353 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.770511 24326 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.875919 21698 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:33.880148 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:33.889526 24337 consensus_meta.cc:287] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.798s user 0.010s sys 0.001s
I20250114 08:59:33.889688 21596 leader_election.cc:403] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:33.876559 21520 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.904649 21696 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:33.905390 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:33.868309 24071 raft_consensus.cc:2743] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:33.909089 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:33.861070 24199 raft_consensus.cc:3049] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:33.934652 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:33.872653 24025 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.878939 24327 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:33.878691 24208 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:33.875797 24382 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250114 08:59:33.924942 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 48501.1us
I20250114 08:59:33.981554 24382 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:33.982127 24382 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:33.983453 24284 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:33.984117 24284 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:33.985047 24284 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:59:33.987643 24325 consensus_queue.cc:1035] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:33.988785 24271 consensus_queue.cc:1046] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 08:59:33.989928 24271 consensus_queue.cc:1225] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 8.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250114 08:59:33.990402 24377 raft_consensus.cc:3049] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Stepping down as leader of term 9
I20250114 08:59:33.990765 24377 raft_consensus.cc:738] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:33.993623 21623 raft_consensus.cc:1270] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250114 08:59:33.911890 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 53635.6us
I20250114 08:59:33.938257 21627 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:33.915625 24272 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:33.943490 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 86308us
I20250114 08:59:33.929930 24372 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:33.918905 24033 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:33.919211 24243 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20250114 08:59:34.000066 21696 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:33.906001 24175 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:33.940889 24253 raft_consensus.cc:2743] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 4. Current term is 5.
I20250114 08:59:33.937345 24199 raft_consensus.cc:738] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:33.909036 24287 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20250114 08:59:33.998543 21669 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:33.934672 24375 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:33.954540 23801 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:33.961658 24337 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.005122 21596 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:34.005659 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 91990us
I20250114 08:59:34.006788 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:34.006101 24365 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.320s user 0.001s sys 0.000s
I20250114 08:59:34.014313 21549 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:33.918254 24381 consensus_queue.cc:1046] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20250114 08:59:34.016156 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:34.009959 24369 log.cc:926] Time spent T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.266s user 0.000s sys 0.001s
I20250114 08:59:34.020273 21701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:46398
I20250114 08:59:34.022418 21701 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:34.022523 21669 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.020092 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.022728 21701 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:33.938292 24352 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.024641 21596 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:34.026718 21521 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:33.991601 24377 consensus_queue.cc:260] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.3, Last appended by leader: 3, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.014320 24316 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.023998 24071 raft_consensus.cc:2743] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:34.013743 24284 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20250114 08:59:34.058593 21549 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:34.060106 21594 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.056728 24377 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.070932 24264 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.027733 24272 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.034962 23801 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:34.044867 24199 consensus_queue.cc:260] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.054513 24372 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:34.086709 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.087393 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.092219 21546 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:34.093163 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.087615 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.096575 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.094488 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.087975 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.090143 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.062353 21701 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.101562 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1895) took 1131 ms. Trace:
I20250114 08:59:34.102034 21699 rpcz_store.cc:276] 0114 08:59:32.969812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:33.008729 (+ 38917us) service_pool.cc:225] Handling call
0114 08:59:33.009132 (+ 403us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:33.328352 (+319220us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:34.101333 (+772981us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b54003705d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:34.101499 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":714328320}
I20250114 08:59:34.105105 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.106549 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.107261 21555 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:34.108947 21553 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:34.099942 21548 raft_consensus.cc:2371] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6. Re-sending same reply.
I20250114 08:59:34.110563 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.111505 21548 raft_consensus.cc:2388] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:34.067637 24166 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.112390 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:34.091166 24268 consensus_meta.cc:287] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.523s user 0.007s sys 0.006s
I20250114 08:59:34.113895 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:34.121910 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 97313.6us
I20250114 08:59:34.122886 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.123127 21596 leader_election.cc:400] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:34.108248 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.124295 21596 leader_election.cc:403] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250114 08:59:34.091022 24356 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:34.118570 21701 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:34.118736 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:34.102093 24383 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20250114 08:59:34.045123 24386 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.129668 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.131518 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.132748 21596 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.134341 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.093372 24372 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:34.127643 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.123960 21546 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:34.129356 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 106665us
I20250114 08:59:34.135242 24272 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:34.141589 24071 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
W20250114 08:59:34.116791 24392 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250114 08:59:34.216261 21549 raft_consensus.cc:1270] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:34.103739 24316 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:34.125062 24199 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.184721 24353 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.029943 24390 consensus_queue.cc:1035] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20250114 08:59:34.227101 21549 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250114 08:59:34.226305 24316 raft_consensus.cc:695] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:34.233345 24316 consensus_queue.cc:237] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.129385 24268 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:34.199021 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 164853us
I20250114 08:59:34.241998 24189 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.243594 24342 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:34.244055 24342 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.245366 24166 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.223242 24351 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.221617 24264 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.252560 21551 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 08:59:34.252859 21553 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:34.253883 24351 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.224393 24175 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.255225 24400 consensus_queue.cc:1035] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250114 08:59:34.256830 24352 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.142735 21548 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.101480 24315 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.260128 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.117010 24396 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:34.116770 23801 raft_consensus.cc:695] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:34.182341 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.263104 21549 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 08:59:34.262516 21552 raft_consensus.cc:2371] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7. Re-sending same reply.
I20250114 08:59:34.192370 24253 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.184041 21596 leader_election.cc:400] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8.
I20250114 08:59:34.205080 21669 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:34.118911 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 127010us
W20250114 08:59:34.118188 24385 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.150s user 0.000s sys 0.002s
I20250114 08:59:34.181550 24372 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.220278 21546 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:34.186134 24326 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.202402 24025 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:34.268321 21621 raft_consensus.cc:2388] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:34.270674 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 87081.3us
I20250114 08:59:34.273289 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.278028 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:34.278861 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 278799us
W20250114 08:59:34.279851 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 74840.5us
I20250114 08:59:34.293371 21551 raft_consensus.cc:3049] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:34.293679 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.281785 24377 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:34.293725 21551 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:34.294559 21668 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.282905 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.296444 21546 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:34.295310 21551 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:34.272598 24346 consensus_meta.cc:287] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.610s user 0.008s sys 0.004s
I20250114 08:59:34.298233 21669 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.298377 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.298996 21551 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:34.304688 24315 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:34.305186 24315 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.305166 21669 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.305747 24315 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.306685 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.307591 24243 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:34.307930 24243 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:34.308151 24243 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.306890 24400 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:34.309125 24400 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.309731 24400 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:34.314628 24243 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:34.315951 22239 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.654s (SENT) [suppressed 37 similar messages]
I20250114 08:59:34.297230 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.316749 24352 consensus_queue.cc:1046] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:34.317342 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.318333 24175 raft_consensus.cc:3049] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 08:59:34.319185 24175 raft_consensus.cc:738] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:34.319245 24315 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.319672 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.320817 24274 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:34.321172 24274 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.321529 24274 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:34.322325 21625 raft_consensus.cc:1270] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 08:59:34.322850 24315 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.323976 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.324858 21625 raft_consensus.cc:2405] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20250114 08:59:34.320137 24175 consensus_queue.cc:260] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.325672 24175 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:34.327157 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.327896 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.328578 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.329108 21624 raft_consensus.cc:2405] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250114 08:59:34.329260 21697 raft_consensus.cc:2363] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:34.329389 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.330060 21698 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:34.330857 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.331465 21626 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:34.318614 24243 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.332258 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.334574 21552 raft_consensus.cc:2388] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:34.333284 21549 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:34.334918 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.335541 21628 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:34.349123 21546 raft_consensus.cc:2388] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:34.350276 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.351042 21624 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.330992 21623 pending_rounds.cc:77] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 1
I20250114 08:59:34.352118 21623 pending_rounds.cc:99] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 08:59:34.352495 21623 raft_consensus.cc:2883] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:34.357766 21596 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.267627 23801 consensus_queue.cc:237] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.368688 21546 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:34.369030 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1447) took 1059 ms. Trace:
I20250114 08:59:34.369814 21669 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.373443 21596 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.332952 24396 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:34.379194 21551 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:34.381845 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.382771 21628 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:34.334345 24189 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.331750 24401 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:34.266222 24382 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.375300 21629 rpcz_store.cc:276] 0114 08:59:33.309970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:33.310095 (+ 125us) service_pool.cc:225] Handling call
0114 08:59:33.310457 (+ 362us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 08:59:33.328715 (+ 18258us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:33.328742 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 08:59:33.332409 (+ 3667us) log.cc:844] Serialized 78 byte log entry
0114 08:59:33.525009 (+192600us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:33.637134 (+112125us) log.cc:844] Serialized 14 byte log entry
0114 08:59:33.672486 (+ 35352us) log.cc:844] Serialized 14 byte log entry
0114 08:59:33.857278 (+184792us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:33.857352 (+ 74us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:34.212559 (+355207us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x7b1800354d00. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:34.256208 (+ 43649us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b1800354d00. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c6947e8 00007fa63c697cc1 00007fa63c697c79 00007fa63c697c09 00007fa63c697bd1 00007fa63c696ecd 00007fa6456d1234 00007fa6456d1069 00007fa63c676fe6
0114 08:59:34.368794 (+112586us) raft_consensus.cc:1680] finished
0114 08:59:34.368818 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:34.368969 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18317,"prepare.queue_time_us":754583,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":120560,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":36241,"spinlock_wait_cycles":34641024,"thread_start_us":398578,"threads_started":3,"wal-append.queue_time_us":319226}
I20250114 08:59:34.349041 24377 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:34.384187 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.400504 21625 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.384235 21669 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.401906 24326 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:34.402280 24326 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:34.402639 24326 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:34.370606 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:34.348824 24253 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:34.381943 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 45085.2us
I20250114 08:59:34.379588 24396 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.420636 24396 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.386605 21596 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.431464 21554 raft_consensus.cc:2363] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
I20250114 08:59:34.394917 24181 raft_consensus.cc:2743] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:34.408236 24377 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:34.391283 24167 consensus_meta.cc:287] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.516s user 0.011s sys 0.000s
I20250114 08:59:34.420323 21624 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:34.447054 21669 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.447624 21699 raft_consensus.cc:1270] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250114 08:59:34.448019 21699 pending_rounds.cc:77] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 3
I20250114 08:59:34.391604 24268 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.423252 22269 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:34.412213 21628 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:34.377470 24315 consensus_queue.cc:1035] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250114 08:59:34.394462 24401 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.448267 21699 pending_rounds.cc:99] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250114 08:59:34.397959 24386 raft_consensus.cc:2758] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20250114 08:59:34.432266 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 45772.4us
I20250114 08:59:34.464443 21596 leader_election.cc:400] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:34.464843 21596 leader_election.cc:403] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:34.459268 21699 raft_consensus.cc:2883] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:34.465790 21669 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.439813 24025 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.468317 21596 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.479063 24342 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.452931 24337 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.481371 21555 raft_consensus.cc:1235] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:34.445587 24264 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.456055 24243 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:34.491448 24243 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:34.492383 24243 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.421041 24253 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.464133 24401 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.484939 21596 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.487323 24167 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6.
I20250114 08:59:34.487871 24268 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:34.489547 24356 raft_consensus.cc:2743] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 08:59:34.486977 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 171040us
I20250114 08:59:34.460686 21625 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 10.
I20250114 08:59:34.492630 22274 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250114 08:59:34.508041 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 202580us
I20250114 08:59:34.508247 21704 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:59:34.495196 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:34.505875 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.507265 24342 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.507378 24268 raft_consensus.cc:2743] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.507558 24272 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:34.514003 24272 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:34.515195 24272 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.493077 24382 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:34.499068 21596 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:34.518764 24268 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:34.519279 24268 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.519757 24268 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:34.506181 21669 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.522037 24386 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:34.522583 24386 raft_consensus.cc:695] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:34.520182 21596 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.522857 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.523522 21696 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.524813 24167 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.527273 21596 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.492447 21548 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 10.
I20250114 08:59:34.532735 21594 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:34.533306 21596 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:34.552577 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 43911.2us
I20250114 08:59:34.553375 21521 leader_election.cc:400] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:34.553740 21521 leader_election.cc:403] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20250114 08:59:34.554399 21596 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.558429 21521 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:34.561810 21696 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 10.
I20250114 08:59:34.567066 22241 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:34.569942 21521 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:34.572120 21596 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.574443 21596 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.575800 24334 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:34.577374 21596 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.578408 24334 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:34.581542 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.582402 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.583321 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.583225 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.494273 24274 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.589057 24274 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.557130 24181 consensus_queue.cc:1035] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250114 08:59:34.594570 24124 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.550272 24167 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:34.495194 24253 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.557410 24188 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:34.604818 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.605424 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.559705 24030 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.495682 24343 consensus_queue.cc:1035] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20250114 08:59:34.571723 24075 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:34.586288 24025 raft_consensus.cc:2743] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:34.596211 24370 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:34.637141 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.586078 24297 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 08:59:34.630368 21700 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:34.617275 24395 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:34.565707 24316 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20250114 08:59:34.644698 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.645730 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.523468 24386 consensus_queue.cc:237] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.625567 24253 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:34.642850 24370 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:34.648881 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.690938 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:34.707019 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 406878us
I20250114 08:59:34.711846 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:34.698639 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.699111 21548 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:34.633972 24075 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:34.715162 21668 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:34.716681 21697 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:34.651630 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.718384 21668 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.719837 21698 raft_consensus.cc:2363] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9.
I20250114 08:59:34.719017 21696 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 10
I20250114 08:59:34.720847 21668 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.721774 21596 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.637881 24188 raft_consensus.cc:695] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:34.696518 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.723166 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.721060 21550 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:34.724789 21668 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.726675 21552 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:34.728094 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:34.696648 24094 consensus_meta.cc:287] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.978s user 0.000s sys 0.013s
I20250114 08:59:34.732398 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.651772 24403 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:34.667860 24400 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.647704 24334 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:34.652513 24025 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:34.753324 24025 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.738509 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.726020 24284 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:34.731388 24395 raft_consensus.cc:695] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:34.728209 24418 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:34.773204 21695 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:34.741009 24272 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250114 08:59:34.739243 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 172297us
I20250114 08:59:34.678380 24297 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:34.710908 24370 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:34.695526 24377 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:34.743647 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.786715 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.722725 24253 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.732061 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.745378 24403 raft_consensus.cc:695] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [term 10 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:34.724745 24075 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:34.793327 21698 raft_consensus.cc:2388] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 10: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:34.744133 21596 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.729096 24310 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:34.803818 21549 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:34.804641 21549 raft_consensus.cc:604] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.739573 24094 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.751958 21699 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:34.737596 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.813838 21704 raft_consensus.cc:2388] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:34.754393 24427 consensus_queue.cc:1046] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20250114 08:59:34.757295 24334 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.780752 24397 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:34.758440 24188 consensus_queue.cc:237] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:34.741813 24298 consensus_meta.cc:287] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.821s user 0.007s sys 0.004s
I20250114 08:59:34.741777 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:34.843526 21552 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:34.737568 24167 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250114 08:59:34.744639 24166 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:34.764649 24395 consensus_queue.cc:237] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.765857 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.805835 24071 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:34.770859 24372 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.744303 21696 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 10.
W20250114 08:59:34.795643 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 51585.8us
I20250114 08:59:34.851603 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.851891 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1433) took 1617 ms. Trace:
I20250114 08:59:34.722668 24352 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:34.748425 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:34.859385 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 111054us
I20250114 08:59:34.860191 21668 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.805159 21549 raft_consensus.cc:612] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:34.806722 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.759663 24284 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:34.780869 24391 log.cc:926] Time spent T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250114 08:59:34.757809 24243 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:34.846975 24375 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:34.874765 24375 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:34.818781 24429 raft_consensus.cc:2758] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250114 08:59:34.787520 24396 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.819581 24268 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.821769 24334 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:34.823921 24030 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:34.780737 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:34.824263 24401 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.782087 22269 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:34.852372 24271 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:34.826287 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.827384 24094 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.831166 21699 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250114 08:59:34.834303 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.774003 24354 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:34.782317 24400 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:34.953197 21627 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:34.728710 24382 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250114 08:59:34.953869 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:34.844609 24397 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:34.839733 24025 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.848512 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:34.851728 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:34.955868 21623 raft_consensus.cc:2405] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:34.791639 21697 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:34.852216 21622 rpcz_store.cc:276] 0114 08:59:33.234987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:33.236746 (+ 1759us) service_pool.cc:225] Handling call
0114 08:59:33.268209 (+ 31463us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:33.563645 (+295436us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:34.851665 (+1288020us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b5400291dd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:34.851838 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":657478400}
I20250114 08:59:34.853108 21596 leader_election.cc:400] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9.
W20250114 08:59:34.957015 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 104058us
I20250114 08:59:34.853986 24181 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250114 08:59:34.857695 24426 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20250114 08:59:34.858027 24372 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:34.863003 21625 raft_consensus.cc:1235] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:34.872903 21551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:34.873312 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.970026 21551 raft_consensus.cc:604] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.970357 21624 raft_consensus.cc:2405] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250114 08:59:34.868528 24352 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:34.880790 24403 consensus_queue.cc:237] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.938454 22271 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 9
I20250114 08:59:34.793872 24297 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.867024 24284 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:34.867142 24175 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:34.980494 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:34.981222 21696 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:34.982123 21696 raft_consensus.cc:1270] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 9 index: 3. Preceding OpId from leader: term: 10 index: 4. (index mismatch)
I20250114 08:59:34.950925 21701 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:34.959197 21704 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:34.966001 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:34.966092 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.967026 21596 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.784545 24418 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.970453 21551 raft_consensus.cc:612] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250114 08:59:34.974370 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 114249us
I20250114 08:59:34.975162 21702 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:34.966353 24397 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:34.847347 24298 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:34.985273 21596 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.003438 21596 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.003800 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.004428 21623 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:35.004925 21596 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:34.991577 21668 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:34.994935 21552 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:34.985872 21667 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:34.997507 21521 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:34.968899 24326 consensus_meta.cc:287] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.566s user 0.006s sys 0.006s
I20250114 08:59:34.999411 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.013841 21624 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:35.009028 21668 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.010234 21594 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.016402 21520 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.018206 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:34.986387 24181 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:35.020623 21520 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.014664 21698 raft_consensus.cc:2949] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } } }
I20250114 08:59:35.011966 24243 raft_consensus.cc:2743] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:34.975925 24272 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250114 08:59:35.026057 21520 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.026427 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.027200 21628 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:35.028088 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.030736 21520 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.007819 24025 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:35.005978 24354 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:35.036545 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.055474 24297 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:35.037336 21668 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.057315 21668 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:35.053261 24274 consensus_queue.cc:1046] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20250114 08:59:35.057854 24315 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:35.058414 24315 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.059365 24315 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.060660 24356 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:35.060864 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.061223 24356 raft_consensus.cc:695] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.061765 24325 consensus_queue.cc:1046] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:35.062122 24356 consensus_queue.cc:237] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.064201 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.043351 21624 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:35.068354 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.069617 21624 raft_consensus.cc:2405] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20250114 08:59:35.064855 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.071844 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.080410 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.034194 21623 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:35.035331 24344 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.012720 24124 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:35.024978 24253 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.047667 22273 raft_consensus.cc:1235] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:35.019656 24199 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:35.049347 24298 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.050879 24359 raft_consensus.cc:3049] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:35.052590 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.053444 24386 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.144829 24372 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:35.130329 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:35.195861 24432 log.cc:926] Time spent T 0205fa1ad333433a8a509e5ad9620de8 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20250114 08:59:35.134837 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.239176 21623 raft_consensus.cc:2371] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10. Re-sending same reply.
I20250114 08:59:35.133661 21548 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 3.
I20250114 08:59:35.022951 24316 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.020923 24336 raft_consensus.cc:2743] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:35.011754 24094 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.036834 24322 raft_consensus.cc:2743] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.178205 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:35.247795 24435 log.cc:926] Time spent T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250114 08:59:35.247848 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.248775 21628 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:35.038174 24030 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.035482 24188 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:35.062139 24343 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.052659 24175 consensus_queue.cc:1046] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
W20250114 08:59:35.209877 21630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1439) took 1965 ms (client timeout 2218 ms). Trace:
I20250114 08:59:35.013538 24326 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.265501 24400 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:35.265997 24188 raft_consensus.cc:695] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:35.266454 24326 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.266099 24400 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.136823 24184 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.267347 24188 consensus_queue.cc:237] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.268127 24184 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.221578 22271 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:35.020668 24397 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.129s
I20250114 08:59:35.035986 24334 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.164s
I20250114 08:59:35.269407 24397 consensus_queue.cc:1225] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 9.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250114 08:59:35.216794 24181 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:35.265066 21630 rpcz_store.cc:269] 0114 08:59:33.244692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:33.246729 (+ 2037us) service_pool.cc:225] Handling call
0114 08:59:35.209572 (+1962843us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54000c9c54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:35.209781 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306720128}
I20250114 08:59:35.270869 24181 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.271571 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.272974 24181 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.020788 24352 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:35.240554 24372 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.275071 24372 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.273168 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.145735 24382 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.276345 24382 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.236948 21622 raft_consensus.cc:2363] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:35.277977 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.278357 24382 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.145187 24124 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:35.279027 24124 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:35.279414 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.280319 21622 raft_consensus.cc:2363] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:35.281862 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.282666 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.282982 21622 raft_consensus.cc:2371] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6. Re-sending same reply.
I20250114 08:59:35.283380 21695 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 10.
I20250114 08:59:35.195535 24319 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:35.245476 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:35.063745 24399 consensus_queue.cc:1035] T a5d394e52b4f4ee2be4e4f4a32ce8547 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.167s
I20250114 08:59:35.228072 24359 raft_consensus.cc:738] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:35.139575 24377 consensus_queue.cc:1046] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20250114 08:59:35.224725 24253 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:35.233213 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:35.252324 24094 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.131672 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.299499 21628 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:35.300662 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.273205 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.302346 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.276084 21630 raft_consensus.cc:2371] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5. Re-sending same reply.
I20250114 08:59:35.225692 24268 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.252280 21552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:35.311221 21552 raft_consensus.cc:604] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.311731 21552 raft_consensus.cc:612] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:35.284291 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:35.242328 24396 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.290504 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.230562 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.285120 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.275487 24184 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.315420 21703 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:35.319757 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.301335 21624 raft_consensus.cc:2371] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8. Re-sending same reply.
I20250114 08:59:35.314286 21622 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:35.321792 21624 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250114 08:59:35.305274 21628 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:35.331487 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.336288 21551 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:35.339900 21520 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.341799 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.343588 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.240752 24386 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.305073 24401 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.355334 21623 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 08:59:35.284260 24319 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.329998 24352 raft_consensus.cc:3049] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:35.359014 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.374773 21520 leader_election.cc:400] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:35.348481 21622 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [4.7-6.8]
I20250114 08:59:35.376353 21623 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:35.344596 24030 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.377925 21669 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.377583 21520 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.379470 21552 raft_consensus.cc:1235] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250114 08:59:35.381242 21628 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250114 08:59:35.356246 24343 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:35.311828 24253 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.386128 21669 leader_election.cc:400] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:35.386636 21669 leader_election.cc:403] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250114 08:59:35.391342 21520 leader_election.cc:400] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10.
I20250114 08:59:35.393155 21520 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.399843 21669 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.400266 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.402648 21596 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.405210 22241 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.406608 24291 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.376039 24352 raft_consensus.cc:738] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.408116 21596 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.412225 21596 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.407554 21552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:35.413905 21552 raft_consensus.cc:604] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.410593 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.387398 24401 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:35.422348 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:35.419322 24316 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:35.399266 24395 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:35.426009 24316 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:35.426419 24316 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:35.428186 24124 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.430593 24124 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.386333 24199 raft_consensus.cc:695] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.438192 21623 raft_consensus.cc:1235] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:35.438522 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.440232 21622 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:35.407385 24399 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:35.412098 24243 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:35.417876 24297 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250114 08:59:35.454272 24440 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250114 08:59:35.454304 24401 raft_consensus.cc:695] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:35.455906 24442 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250114 08:59:35.455636 24401 consensus_queue.cc:237] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.457480 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.458305 21627 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:35.448539 24310 consensus_meta.cc:287] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.634s user 0.003s sys 0.008s
I20250114 08:59:35.411131 24400 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:35.359501 24359 consensus_queue.cc:260] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.462623 21627 raft_consensus.cc:1235] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20250114 08:59:35.463364 24370 consensus_meta.cc:287] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.651s user 0.009s sys 0.002s
I20250114 08:59:35.412926 24445 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.452925 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.468114 21626 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
W20250114 08:59:35.469869 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 51050.5us
I20250114 08:59:35.435699 24268 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:35.494545 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 107774us
I20250114 08:59:35.447060 24343 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250114 08:59:35.496857 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 44313.6us
I20250114 08:59:35.512859 24030 consensus_queue.cc:1046] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:35.514506 24030 consensus_queue.cc:1046] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:35.514899 24310 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.515846 24030 raft_consensus.cc:3049] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:35.515712 24310 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.516986 24448 raft_consensus.cc:3049] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:35.517539 24448 raft_consensus.cc:738] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:35.518509 24310 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.518648 24448 consensus_queue.cc:260] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.473502 24243 raft_consensus.cc:695] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.473928 24319 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.493425 24167 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.465178 23801 consensus_queue.cc:1046] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20250114 08:59:35.432924 24199 consensus_queue.cc:237] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.540746 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:35.541046 21621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1455) took 2151 ms (client timeout 2433 ms). Trace:
I20250114 08:59:35.541510 21623 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 10.
W20250114 08:59:35.541713 21621 rpcz_store.cc:269] 0114 08:59:33.389534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:33.391037 (+ 1503us) service_pool.cc:225] Handling call
0114 08:59:35.540816 (+2149779us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b54002913d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:35.540957 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1673902464}
I20250114 08:59:35.542405 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.543593 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1491) took 1455 ms. Trace:
I20250114 08:59:35.543716 21621 raft_consensus.cc:1270] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20250114 08:59:35.543946 21554 rpcz_store.cc:276] 0114 08:59:34.087878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.088136 (+ 258us) service_pool.cc:225] Handling call
0114 08:59:35.543444 (+1455308us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b540008d254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:35.543541 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679438080}
I20250114 08:59:35.544343 21621 pending_rounds.cc:77] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 3
I20250114 08:59:35.544801 21621 pending_rounds.cc:99] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 7.4
W20250114 08:59:35.545109 21550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 190) took 2580 ms (client timeout 1654 ms). Trace:
I20250114 08:59:35.545167 21621 raft_consensus.cc:2883] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:35.546037 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:35.545744 21550 rpcz_store.cc:269] 0114 08:59:32.965225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:32.967117 (+ 1892us) service_pool.cc:225] Handling call
0114 08:59:35.544924 (+2577807us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540007b7d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:35.545039 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3986646784}
I20250114 08:59:35.547019 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.547271 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:35.547614 21550 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:35.522900 24336 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:35.488517 24382 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20250114 08:59:35.480818 21622 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:35.553618 21547 raft_consensus.cc:1270] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250114 08:59:35.557794 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.559652 21596 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.574316 21668 leader_election.cc:400] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
I20250114 08:59:35.516408 24030 raft_consensus.cc:738] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:35.579258 24030 consensus_queue.cc:260] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.581146 24030 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:35.522490 24124 consensus_queue.cc:1046] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20250114 08:59:35.575033 24399 consensus_queue.cc:1046] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20250114 08:59:35.585227 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.585783 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.575625 21668 leader_election.cc:403] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:35.547654 21621 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:35.578127 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:35.585310 21627 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:55318
I20250114 08:59:35.587845 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.587054 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.549903 24167 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.575677 21596 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.589382 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.590436 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46402 (request call id 1471) took 1008 ms. Trace:
I20250114 08:59:35.591028 21702 rpcz_store.cc:276] 0114 08:59:34.581761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.582088 (+ 327us) service_pool.cc:225] Handling call
0114 08:59:35.580665 (+998577us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54001c0f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:35.590360 (+ 9695us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":867156736}
I20250114 08:59:35.590889 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.597421 21550 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:35.600338 21669 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.601995 21594 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:35.603565 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:35.591845 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.605798 21621 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:35.606541 21596 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.534111 24094 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:35.494701 24370 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.607807 21623 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 4. Current term is 6. Ops: [4.4-4.4]
I20250114 08:59:35.608691 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:35.551143 24448 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:35.625754 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:35.572049 24284 consensus_meta.cc:287] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.586s user 0.010s sys 0.000s
I20250114 08:59:35.573019 24336 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:35.488178 24352 consensus_queue.cc:260] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.610960 24372 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.623159 21596 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.573688 24188 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:35.624825 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.633412 21550 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250114 08:59:35.624739 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.572661 24452 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.585265 24399 raft_consensus.cc:3049] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:35.623365 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:35.645727 21703 pending_rounds.cc:77] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 2
I20250114 08:59:35.660363 21703 pending_rounds.cc:99] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 9.3
I20250114 08:59:35.660775 21703 raft_consensus.cc:2883] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:35.647241 21700 raft_consensus.cc:1270] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250114 08:59:35.640690 24396 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.610201 24325 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
I20250114 08:59:35.622438 24297 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:35.653293 24253 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.618567 24389 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:35.607784 24386 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.602983 24310 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.665524 21623 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:35.643090 24181 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.655356 24336 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:35.628518 23801 consensus_queue.cc:1035] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
W20250114 08:59:35.669714 24025 consensus_meta.cc:287] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.597s user 0.005s sys 0.005s
W20250114 08:59:35.652052 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 99679.5us
I20250114 08:59:35.646898 24284 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.699126 24316 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.659380 24399 raft_consensus.cc:738] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.701196 21700 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20250114 08:59:35.656318 24243 consensus_queue.cc:237] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:35.679836 24075 consensus_meta.cc:287] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.865s user 0.010s sys 0.001s
I20250114 08:59:35.731770 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:35.761632 22270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49020 (request call id 290) took 1231 ms. Trace:
I20250114 08:59:35.733582 22273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 286) took 1771 ms. Trace:
W20250114 08:59:35.704037 24093 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.575s user 0.015s sys 0.015s
I20250114 08:59:35.740988 21627 raft_consensus.cc:604] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.730899 24372 raft_consensus.cc:2743] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:35.768045 21627 raft_consensus.cc:612] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:35.741847 21621 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:35.739431 21668 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:35.681167 24297 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:35.769186 24297 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:35.752002 21623 raft_consensus.cc:1270] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20250114 08:59:35.772643 21623 pending_rounds.cc:77] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 4
I20250114 08:59:35.772910 21623 pending_rounds.cc:99] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20250114 08:59:35.773144 21623 raft_consensus.cc:2883] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:35.773810 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1688) took 1055 ms. Trace:
I20250114 08:59:35.774013 21702 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:35.774134 21623 rpcz_store.cc:276] 0114 08:59:34.717938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.726367 (+1008429us) service_pool.cc:225] Handling call
0114 08:59:35.751634 (+ 25267us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.773572 (+ 21938us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.773751 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48128}
W20250114 08:59:35.775789 24375 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.900s user 0.013s sys 0.000s
I20250114 08:59:35.775952 21623 raft_consensus.cc:1270] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:35.752226 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1686) took 1034 ms. Trace:
I20250114 08:59:35.741900 24025 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.764434 22273 rpcz_store.cc:276] 0114 08:59:33.961850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:33.965747 (+ 3897us) service_pool.cc:225] Handling call
0114 08:59:34.024648 (+ 58901us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.225746 (+1201098us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.733197 (+507451us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400599f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:35.733488 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2288135424}
I20250114 08:59:35.762135 22270 rpcz_store.cc:276] 0114 08:59:34.530480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.530647 (+ 167us) service_pool.cc:225] Handling call
0114 08:59:35.731524 (+1200877us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:35.761543 (+ 30019us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5273856}
I20250114 08:59:35.764221 24075 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:35.781415 24456 log.cc:926] Time spent T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20250114 08:59:35.782611 24075 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.783425 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 298) took 1093 ms. Trace:
I20250114 08:59:35.783803 21624 rpcz_store.cc:276] 0114 08:59:34.690397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.330066 (+639669us) service_pool.cc:225] Handling call
0114 08:59:35.330388 (+ 322us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:35.333776 (+ 3388us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.333822 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:35.335365 (+ 1543us) log.cc:844] Serialized 26 byte log entry
0114 08:59:35.439288 (+103923us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:35.439325 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.441508 (+ 2183us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:35.783178 (+341670us) raft_consensus.cc:1680] finished
0114 08:59:35.783207 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:35.783369 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4589,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":4535,"thread_start_us":58454,"threads_started":1,"wal-append.queue_time_us":273255}
I20250114 08:59:35.784564 24399 consensus_queue.cc:260] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.785979 24284 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.786260 21624 raft_consensus.cc:1270] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 10 index: 4. (index mismatch)
I20250114 08:59:35.786391 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1473) took 1454 ms. Trace:
W20250114 08:59:35.765779 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 191982us
I20250114 08:59:35.787550 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1692) took 1068 ms. Trace:
W20250114 08:59:35.765331 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 142297us
I20250114 08:59:35.788300 21624 rpcz_store.cc:276] 0114 08:59:34.718920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.785649 (+1066729us) service_pool.cc:225] Handling call
0114 08:59:35.785941 (+ 292us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.787312 (+ 1371us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.787491 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:35.789526 24418 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:35.786775 21555 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:35.664053 24188 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.737152 24389 raft_consensus.cc:695] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:35.776607 21622 rpcz_store.cc:276] 0114 08:59:34.717462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.726151 (+1008689us) service_pool.cc:225] Handling call
0114 08:59:35.751634 (+ 25483us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.751845 (+ 211us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.751879 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:35.751929 (+ 50us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:35.751973 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.752027 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:35.752160 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39680}
W20250114 08:59:35.786903 24093 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.661s user 0.015s sys 0.016s
I20250114 08:59:35.801877 24075 consensus_queue.cc:1046] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 08:59:35.802821 24175 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:35.803088 24075 raft_consensus.cc:3049] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:35.803633 24075 raft_consensus.cc:738] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:35.787539 21629 rpcz_store.cc:276] 0114 08:59:34.331903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.406676 (+ 74773us) service_pool.cc:225] Handling call
0114 08:59:34.407136 (+ 460us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:34.415320 (+ 8184us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:34.440517 (+ 25197us) log.cc:844] Serialized 14 byte log entry
0114 08:59:35.448628 (+1008111us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b18003d71a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fe2b 00007fa63f5d797a
0114 08:59:35.739962 (+291334us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:35.740588 (+ 626us) log.cc:844] Serialized 26 byte log entry
0114 08:59:35.740774 (+ 186us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:35.740816 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.740931 (+ 115us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:35.785440 (+ 44509us) raft_consensus.cc:1680] finished
0114 08:59:35.785589 (+ 149us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:35.786305 (+ 716us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17130,"prepare.queue_time_us":24961,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":74731,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":27965,"spinlock_wait_cycles":11684096,"thread_start_us":275675,"threads_started":2,"wal-append.queue_time_us":1250224}
I20250114 08:59:35.791714 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 299) took 1060 ms. Trace:
W20250114 08:59:35.795728 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 56415.6us
I20250114 08:59:35.794657 21596 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.786075 24399 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:35.816588 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 49676.1us
I20250114 08:59:35.818385 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.821108 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:35.784018 21702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46386 (request call id 224) took 1143 ms. Trace:
I20250114 08:59:35.802054 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.826596 21521 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.803191 24395 raft_consensus.cc:3049] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:35.794796 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1483) took 1022 ms. Trace:
I20250114 08:59:35.799993 21554 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:35.834662 21621 rpcz_store.cc:276] 0114 08:59:34.772861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.793907 (+1021046us) service_pool.cc:225] Handling call
0114 08:59:35.794322 (+ 415us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.794505 (+ 183us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.794528 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:35.794571 (+ 43us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:35.794603 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.794650 (+ 47us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:35.794739 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:35.682976 24396 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.814443 21551 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:35.815356 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1691) took 1096 ms. Trace:
I20250114 08:59:35.845736 21623 rpcz_store.cc:276] 0114 08:59:34.718645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.775245 (+1056600us) service_pool.cc:225] Handling call
0114 08:59:35.775603 (+ 358us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.776348 (+ 745us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.815290 (+ 38942us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:35.830701 21624 rpcz_store.cc:276] 0114 08:59:34.731146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.789728 (+1058582us) service_pool.cc:225] Handling call
0114 08:59:35.790236 (+ 508us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.790863 (+ 627us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.790888 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:35.790953 (+ 65us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:35.791012 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.791095 (+ 83us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:35.791635 (+ 540us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:35.836113 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.837255 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.833119 21702 rpcz_store.cc:276] 0114 08:59:34.640627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.779583 (+1138956us) service_pool.cc:225] Handling call
0114 08:59:35.783614 (+ 4031us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:35.852424 21621 raft_consensus.cc:2388] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:35.869025 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:35.871204 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 69234.2us
W20250114 08:59:35.875802 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 49322.2us
I20250114 08:59:35.885803 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:35.890241 21596 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:35.824927 24372 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 08:59:35.776294 24375 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.876240 21624 raft_consensus.cc:1235] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:35.834734 24208 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:35.923854 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 44616.8us
I20250114 08:59:35.926905 24372 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.925935 24367 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:35.935353 24367 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:35.937855 24367 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.938000 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:35.834596 24395 raft_consensus.cc:738] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:35.923859 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1487) took 1089 ms. Trace:
I20250114 08:59:35.921988 24093 raft_consensus.cc:2743] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:35.927634 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:35.835019 24124 consensus_queue.cc:1046] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20250114 08:59:35.931249 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:35.860270 24315 raft_consensus.cc:2798] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:35.932250 21695 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:35.933542 24375 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.942622 21623 rpcz_store.cc:276] 0114 08:59:34.834025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.846820 (+1012795us) service_pool.cc:225] Handling call
0114 08:59:35.923165 (+ 76345us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.923400 (+ 235us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.923437 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:35.923493 (+ 56us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:35.923538 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.923623 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:35.923779 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:35.869833 24370 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:35.890491 24465 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250114 08:59:35.931404 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:35.959700 21702 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:35.878407 24322 consensus_queue.cc:1035] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20250114 08:59:35.986181 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:35.986856 21623 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:35.941805 24316 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.985033 21624 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:35.992365 24463 log.cc:926] Time spent T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250114 08:59:35.992837 24465 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.993716 21624 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20250114 08:59:35.993358 24465 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:35.994136 24464 log.cc:926] Time spent T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20250114 08:59:35.995645 24465 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:35.953243 24389 consensus_queue.cc:237] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:35.998590 24284 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:35.999137 24284 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.001279 24284 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:35.815783 24075 consensus_queue.cc:260] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:35.953387 24315 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:36.002087 24315 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:36.002255 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1485) took 1282 ms. Trace:
I20250114 08:59:36.002655 21704 rpcz_store.cc:276] 0114 08:59:34.720042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.829182 (+109140us) service_pool.cc:225] Handling call
0114 08:59:34.958731 (+129549us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:34.983695 (+ 24964us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.002049 (+1018354us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540023f5d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.002193 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":266568064}
I20250114 08:59:36.002828 24395 consensus_queue.cc:260] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.005637 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:36.010605 21551 raft_consensus.cc:1235] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 08:59:35.974062 24310 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:35.976750 21521 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:35.946398 23801 consensus_queue.cc:1035] T 6e6879e06c934d438e3ca9c3f093d4be P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.063s
I20250114 08:59:35.925048 24468 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.133s
W20250114 08:59:36.030452 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 72162.5us
I20250114 08:59:36.035233 24291 consensus_queue.cc:1046] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20250114 08:59:36.045477 24387 raft_consensus.cc:3049] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:36.045904 24387 raft_consensus.cc:738] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:36.047200 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1943) took 1306 ms. Trace:
I20250114 08:59:36.047644 21699 rpcz_store.cc:276] 0114 08:59:34.740636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.950092 (+209456us) service_pool.cc:225] Handling call
0114 08:59:34.953584 (+ 3492us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:34.979563 (+ 25979us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:35.145984 (+166421us) log.cc:844] Serialized 14 byte log entry
0114 08:59:35.240059 (+ 94075us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:35.240104 (+ 45us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:35.240152 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.046976 (+806824us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b5400346a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.047018 (+ 42us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.047139 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61251,"prepare.queue_time_us":30636,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":3495,"raft.queue_time_us":61609,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":7683,"spinlock_wait_cycles":59101440,"thread_start_us":137666,"threads_started":1}
I20250114 08:59:36.050073 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1945) took 1287 ms. Trace:
I20250114 08:59:36.050480 21701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1932) took 1447 ms. Trace:
W20250114 08:59:36.051169 21553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1495) took 1961 ms (client timeout 1984 ms). Trace:
W20250114 08:59:36.051577 21553 rpcz_store.cc:269] 0114 08:59:34.090707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.113671 (+ 22964us) service_pool.cc:225] Handling call
0114 08:59:36.051011 (+1937340us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540007d0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.051113 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300445056}
I20250114 08:59:36.053078 24336 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:36.053875 21626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57320 (request call id 1472) took 1722 ms (client timeout 2170 ms). Trace:
W20250114 08:59:36.054309 21626 rpcz_store.cc:269] 0114 08:59:34.331551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.332058 (+ 507us) service_pool.cc:225] Handling call
0114 08:59:36.053705 (+1721647us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540039e754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.053817 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2605967744}
I20250114 08:59:36.055341 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:36.055965 21626 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:36.055799 24264 consensus_queue.cc:1035] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:59:36.057279 21554 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:36.058158 24439 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.801s user 0.002s sys 0.003s
I20250114 08:59:36.021772 24370 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:36.059481 21549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1613) took 1190 ms. Trace:
I20250114 08:59:36.046728 24387 consensus_queue.cc:260] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.061111 21669 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:36.065251 21553 raft_consensus.cc:1235] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 08:59:36.066622 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.067443 21553 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:36.050453 21697 rpcz_store.cc:276] 0114 08:59:34.762733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.957328 (+194595us) service_pool.cc:225] Handling call
0114 08:59:34.957613 (+ 285us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:34.986780 (+ 29167us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:34.986851 (+ 71us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:34.986912 (+ 61us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:34.986963 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.049864 (+1062901us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b54002e9d54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.049903 (+ 39us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.050019 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66470656}
I20250114 08:59:36.059896 21549 rpcz_store.cc:276] 0114 08:59:34.869193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.869401 (+ 208us) service_pool.cc:225] Handling call
0114 08:59:34.869890 (+ 489us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:34.870409 (+ 519us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:34.870453 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:34.871156 (+ 703us) log.cc:844] Serialized 26 byte log entry
0114 08:59:34.971677 (+100521us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:34.971728 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:34.971820 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:36.001134 (+1029314us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:36.059194 (+ 58060us) raft_consensus.cc:1680] finished
0114 08:59:36.059216 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.059422 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":370,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":88,"spinlock_wait_cycles":7891328,"thread_start_us":20989,"threads_started":1,"wal-append.queue_time_us":141076}
I20250114 08:59:36.079231 21669 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:36.081566 21549 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250114 08:59:36.062275 21702 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:36.050925 21701 rpcz_store.cc:276] 0114 08:59:34.603303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.606009 (+ 2706us) service_pool.cc:225] Handling call
0114 08:59:34.606357 (+ 348us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:34.982797 (+376440us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.050273 (+1067476us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b5400328a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.050424 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":743427840}
I20250114 08:59:36.021065 21593 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:36.018522 24372 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20250114 08:59:36.061136 24208 raft_consensus.cc:2743] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:36.063455 24429 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:36.086784 24243 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:36.086184 24284 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:36.025700 24310 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.059918 24370 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.094156 24387 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:36.093014 24297 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:36.093933 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 72970.5us
I20250114 08:59:36.096264 21623 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:36.275619 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 296) took 1770 ms. Trace:
I20250114 08:59:36.276144 21625 rpcz_store.cc:276] 0114 08:59:34.505327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.506254 (+ 927us) service_pool.cc:225] Handling call
0114 08:59:34.555653 (+ 49399us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:34.969737 (+414084us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.274884 (+1305147us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54002de6d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.275346 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":706208000}
I20250114 08:59:36.277148 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 301) took 1015 ms. Trace:
I20250114 08:59:36.278703 21623 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250114 08:59:36.278839 21625 raft_consensus.cc:1270] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20250114 08:59:36.269061 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 229340us
I20250114 08:59:36.282825 21625 raft_consensus.cc:1270] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:36.096305 21596 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:36.262786 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 197) took 2236 ms. Trace:
W20250114 08:59:36.258850 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 418393us
I20250114 08:59:36.286664 21520 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:36.289300 21547 rpcz_store.cc:276] 0114 08:59:34.026001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.026120 (+ 119us) service_pool.cc:225] Handling call
0114 08:59:35.545150 (+1519030us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.576203 (+ 31053us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.261991 (+685788us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540007b7d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.262388 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3492663168}
I20250114 08:59:36.301335 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 297) took 1615 ms. Trace:
I20250114 08:59:36.302013 21628 rpcz_store.cc:276] 0114 08:59:34.685834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.323235 (+637401us) service_pool.cc:225] Handling call
0114 08:59:35.380905 (+ 57670us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.383730 (+ 2825us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.300803 (+917073us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b54001518d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.301100 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72058240}
I20250114 08:59:36.305303 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:36.313849 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 08:59:36.325968 21451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36438 (request call id 75) took 8693 ms. Trace:
I20250114 08:59:36.373006 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:36.376303 21695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1950) took 1401 ms. Trace:
I20250114 08:59:36.376888 21695 rpcz_store.cc:276] 0114 08:59:34.975250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.328882 (+353632us) service_pool.cc:225] Handling call
0114 08:59:35.329377 (+ 495us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.947797 (+618420us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.375587 (+427790us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b5400170054. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.376045 (+ 458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1386051968}
I20250114 08:59:36.326617 21451 rpcz_store.cc:276] 0114 08:59:27.632890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:27.633495 (+ 605us) service_pool.cc:225] Handling call
0114 08:59:36.325635 (+8692140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 08:59:33.141357 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 08:59:33.141894 (+ 537us) write_op.cc:432] Acquiring schema lock in shared mode
0114 08:59:33.141930 (+ 36us) write_op.cc:435] Acquired schema lock
0114 08:59:33.141955 (+ 25us) tablet.cc:659] Decoding operations
0114 08:59:33.142727 (+ 772us) write_op.cc:620] Acquiring the partition lock for write op
0114 08:59:33.142805 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0114 08:59:33.142830 (+ 25us) tablet.cc:682] Acquiring locks for 6 operations
0114 08:59:33.143325 (+ 495us) tablet.cc:698] Row locks acquired
0114 08:59:33.143342 (+ 17us) write_op.cc:260] PREPARE: finished
0114 08:59:33.143491 (+ 149us) write_op.cc:270] Start()
0114 08:59:33.143609 (+ 118us) write_op.cc:276] Timestamp: P: 1736845173143440 usec, L: 0
0114 08:59:33.143628 (+ 19us) op_driver.cc:348] REPLICATION: starting
0114 08:59:33.185364 (+ 41736us) log.cc:844] Serialized 2597 byte log entry
0114 08:59:34.036114 (+850750us) op_driver.cc:464] REPLICATION: finished
0114 08:59:34.455592 (+419478us) write_op.cc:301] APPLY: starting
0114 08:59:34.461954 (+ 6362us) tablet.cc:1364] starting BulkCheckPresence
0114 08:59:34.462136 (+ 182us) tablet.cc:1367] finished BulkCheckPresence
0114 08:59:34.462270 (+ 134us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 08:59:34.586635 (+124365us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 08:59:34.641342 (+ 54707us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0114 08:59:34.646563 (+ 5221us) write_op.cc:312] APPLY: finished
0114 08:59:34.704487 (+ 57924us) log.cc:844] Serialized 52 byte log entry
0114 08:59:34.723382 (+ 18895us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:34.731367 (+ 7985us) write_op.cc:454] Released schema lock
0114 08:59:34.737222 (+ 5855us) write_op.cc:341] FINISH: Updating metrics
0114 08:59:34.775264 (+ 38042us) write_op.cc:489] Releasing partition, row and schema locks
0114 08:59:34.788946 (+ 13682us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":734818,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":409326,"apply.run_cpu_time_us":3870,"apply.run_wall_time_us":300500,"num_ops":6,"prepare.queue_time_us":181966,"prepare.run_cpu_time_us":17973,"prepare.run_wall_time_us":202407,"raft.queue_time_us":147337,"raft.run_cpu_time_us":35381,"raft.run_wall_time_us":419961,"replication_time_us":881745,"thread_start_us":644005,"threads_started":4,"wal-append.queue_time_us":328404,"wal-append.run_cpu_time_us":50515,"wal-append.run_wall_time_us":2224832}]]}
I20250114 08:59:36.405900 22266 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250114 08:59:36.407179 24033 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:36.408746 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 134855us
I20250114 08:59:36.411761 23801 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:36.413899 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 317513us
I20250114 08:59:36.414908 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1486) took 1624 ms. Trace:
I20250114 08:59:36.277761 21626 rpcz_store.cc:276] 0114 08:59:35.261552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.072722 (+811170us) service_pool.cc:225] Handling call
0114 08:59:36.073131 (+ 409us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:36.107720 (+ 34589us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:36.107779 (+ 59us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:36.107838 (+ 59us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:36.107884 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.276482 (+168598us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b540022db54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.276603 (+ 121us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.276924 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78986624}
I20250114 08:59:36.446152 21629 rpcz_store.cc:276] 0114 08:59:34.790720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.806387 (+1015667us) service_pool.cc:225] Handling call
0114 08:59:35.806788 (+ 401us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.807050 (+ 262us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:36.018344 (+211294us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.413840 (+395496us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.414196 (+ 356us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:36.414288 (+ 92us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:36.414407 (+ 119us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.414510 (+ 103us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.414702 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":307344,"prepare.queue_time_us":248553,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":42,"thread_start_us":262666,"threads_started":2,"wal-append.queue_time_us":135697}
I20250114 08:59:36.471712 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:36.482550 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1614) took 1517 ms. Trace:
I20250114 08:59:36.483253 21555 rpcz_store.cc:276] 0114 08:59:34.965483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.965653 (+ 170us) service_pool.cc:225] Handling call
0114 08:59:36.263445 (+1297792us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b5400194e54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.482323 (+218878us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1844953984}
I20250114 08:59:36.483826 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:36.484643 24033 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:36.490947 24470 log.cc:926] Time spent T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.504s user 0.004s sys 0.008s
I20250114 08:59:36.492828 21546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1527) took 1425 ms. Trace:
I20250114 08:59:36.114847 24395 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:36.493372 21546 rpcz_store.cc:276] 0114 08:59:35.067260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.346177 (+278917us) service_pool.cc:225] Handling call
0114 08:59:35.346551 (+ 374us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.384452 (+ 37901us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.384505 (+ 53us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:35.384553 (+ 48us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:35.384588 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.295465 (+910877us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b5400193f54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.435804 (+140339us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.492587 (+ 56783us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86616960}
I20250114 08:59:36.096941 24427 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:36.497756 24427 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.498754 21623 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20250114 08:59:36.488152 24033 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:36.100441 24429 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.501484 23801 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.502686 24264 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:36.099857 24284 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:36.503732 24429 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.504047 24284 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:36.097797 24466 raft_consensus.cc:2743] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250114 08:59:36.137944 24454 log.cc:926] Time spent T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.465s user 0.012s sys 0.011s
I20250114 08:59:36.099557 24243 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:36.507315 24243 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:36.100080 24188 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:36.137682 24315 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.511133 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:36.485096 24344 consensus_queue.cc:1046] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:59:36.133601 24310 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.488179 24353 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:36.110903 24370 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:36.514150 24353 raft_consensus.cc:695] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:36.177768 24398 raft_consensus.cc:3049] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:36.515138 24398 raft_consensus.cc:738] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:36.515937 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:36.517308 21695 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:36.516824 24398 consensus_queue.cc:260] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.520694 21696 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:36.520197 24353 consensus_queue.cc:237] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.520906 21629 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:36.521673 21628 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:36.522439 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1956) took 1110 ms. Trace:
I20250114 08:59:36.522854 21696 rpcz_store.cc:276] 0114 08:59:35.411560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.372121 (+960561us) service_pool.cc:225] Handling call
0114 08:59:36.522238 (+150117us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54001ef0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:36.522358 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":336071936}
I20250114 08:59:36.525087 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:36.525802 24446 consensus_queue.cc:1035] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:36.131616 24297 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:36.530753 24420 consensus_queue.cc:1046] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:36.105696 24396 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:36.533869 24373 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.522855 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:36.544358 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.534813 24373 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.504392 23801 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.571313 21549 raft_consensus.cc:2388] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:36.102803 24336 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.588990 24398 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:36.570608 24272 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:36.602813 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:36.604173 21627 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
W20250114 08:59:36.605253 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 163176us
I20250114 08:59:36.609042 24319 raft_consensus.cc:3049] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:36.609632 24319 raft_consensus.cc:738] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:36.611227 24319 consensus_queue.cc:260] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.615870 24346 raft_consensus.cc:3049] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:36.616365 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1479) took 2109 ms. Trace:
I20250114 08:59:36.616967 21546 raft_consensus.cc:1270] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20250114 08:59:36.618155 21621 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s user 0.005s sys 0.005s
I20250114 08:59:36.616916 21630 rpcz_store.cc:276] 0114 08:59:34.506853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.303754 (+796901us) service_pool.cc:225] Handling call
0114 08:59:35.304171 (+ 417us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:35.313329 (+ 9158us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.313373 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:35.341596 (+ 28223us) log.cc:844] Serialized 26 byte log entry
0114 08:59:35.491727 (+150131us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:35.491763 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:35.491834 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:36.137130 (+645296us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:36.615658 (+478528us) raft_consensus.cc:1680] finished
0114 08:59:36.615703 (+ 45us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.616098 (+ 395us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8660,"raft.queue_time_us":16523,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5821,"spinlock_wait_cycles":67062528,"thread_start_us":118442,"threads_started":1,"wal-append.queue_time_us":308000}
I20250114 08:59:36.618924 24490 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.590258 24427 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.619735 24490 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.620661 24315 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:36.621140 24175 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:36.621542 24315 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:36.622020 24175 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.622649 24490 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.625833 24399 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:36.626673 24399 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.622370 24315 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.629843 22275 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250114 08:59:36.605273 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1484) took 1842 ms. Trace:
W20250114 08:59:36.635059 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57391.6us
I20250114 08:59:36.640817 24468 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.641649 24468 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.642870 21628 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250114 08:59:36.644436 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 132057us
I20250114 08:59:36.644552 24466 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:36.644876 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:36.645641 21623 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:36.645320 24466 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.646140 24167 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.646307 24188 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:36.649395 24025 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.650133 24025 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.651193 21625 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:36.651434 24297 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.651912 24297 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.588776 24325 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:36.640045 24264 raft_consensus.cc:695] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:36.653119 24325 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.653839 24264 consensus_queue.cc:237] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.671540 24272 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.674871 24272 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:36.616425 24346 raft_consensus.cc:738] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:36.735623 21623 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:36.715339 21628 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250114 08:59:36.634539 21698 rpcz_store.cc:276] 0114 08:59:34.719686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.825778 (+106092us) service_pool.cc:225] Handling call
0114 08:59:34.826299 (+ 521us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:34.875204 (+ 48905us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:34.875237 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:34.876038 (+ 801us) log.cc:844] Serialized 508 byte log entry
0114 08:59:35.014557 (+138519us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:35.229623 (+215066us) log.cc:844] Serialized 14 byte log entry
0114 08:59:35.250177 (+ 20554us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.376418 (+1126241us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b5400370fd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:36.419495 (+ 43077us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:36.472898 (+ 53403us) raft_consensus.cc:1680] finished
0114 08:59:36.481354 (+ 8456us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.562065 (+ 80711us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24061,"raft.queue_time_us":34486,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":111580416,"thread_start_us":138248,"threads_started":1,"wal-append.queue_time_us":352227,"wal-append.run_cpu_time_us":2696,"wal-append.run_wall_time_us":1252414}
W20250114 08:59:36.724324 21622 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s user 0.000s sys 0.020s
I20250114 08:59:36.654387 24297 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.694089 24175 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:36.766417 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:36.767352 21630 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:36.763729 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:36.747409 24468 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:36.770078 21624 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.011s sys 0.005s
W20250114 08:59:36.770510 21624 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.011s sys 0.005s
I20250114 08:59:36.771466 24186 consensus_queue.cc:1046] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:36.771693 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1698) took 1709 ms. Trace:
I20250114 08:59:36.772428 21629 raft_consensus.cc:1235] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250114 08:59:36.772768 24186 raft_consensus.cc:3049] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:36.773276 24186 raft_consensus.cc:738] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:36.774487 24186 consensus_queue.cc:260] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.790810 24186 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:36.792955 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1489) took 1803 ms. Trace:
I20250114 08:59:36.793994 24490 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:36.795630 24427 consensus_queue.cc:1046] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:36.796730 24427 raft_consensus.cc:3049] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:36.797171 24427 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:36.801631 24399 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:36.803737 24325 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:36.804986 24373 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.758919 24466 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.798305 24427 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.793464 21621 rpcz_store.cc:276] 0114 08:59:34.989804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.868174 (+878370us) service_pool.cc:225] Handling call
0114 08:59:35.868539 (+ 365us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.868777 (+ 238us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:36.088554 (+219777us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.725047 (+636493us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:36.760038 (+ 34991us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:36.792554 (+ 32516us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.792687 (+ 133us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.792885 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45981,"prepare.queue_time_us":168426,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":41,"raft.queue_time_us":74912,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":43799,"thread_start_us":673022,"threads_started":2,"wal-append.queue_time_us":717552}
I20250114 08:59:36.723001 24025 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.817107 24346 consensus_queue.cc:260] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.818722 24346 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:36.772272 21624 rpcz_store.cc:276] 0114 08:59:35.061887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.996029 (+934142us) service_pool.cc:225] Handling call
0114 08:59:35.996406 (+ 377us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:35.998012 (+ 1606us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:35.998045 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:36.022170 (+ 24125us) log.cc:844] Serialized 26 byte log entry
0114 08:59:36.771084 (+748914us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:36.771187 (+ 103us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.771362 (+ 175us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:36.771445 (+ 83us) raft_consensus.cc:1680] finished
0114 08:59:36.771463 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.771638 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":354,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":687548,"threads_started":1,"wal-append.queue_time_us":575492}
I20250114 08:59:36.821810 24243 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.814450 24427 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:36.827654 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:36.824323 24243 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:36.832726 24319 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:36.837684 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1540) took 1072 ms. Trace:
I20250114 08:59:36.838157 21548 rpcz_store.cc:276] 0114 08:59:35.765191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.046014 (+280823us) service_pool.cc:225] Handling call
0114 08:59:36.046392 (+ 378us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:36.063341 (+ 16949us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:36.063388 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:36.077669 (+ 14281us) log.cc:844] Serialized 26 byte log entry
0114 08:59:36.596438 (+518769us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:36.600290 (+ 3852us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.634289 (+ 33999us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:36.776137 (+141848us) raft_consensus.cc:1680] finished
0114 08:59:36.776155 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.837487 (+ 61332us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16531,"raft.queue_time_us":17036,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7907,"thread_start_us":303569,"threads_started":1,"wal-append.queue_time_us":681177}
I20250114 08:59:36.843529 21596 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:36.846814 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:36.850890 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:36.860993 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 52932.4us
I20250114 08:59:36.867086 21623 raft_consensus.cc:3054] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:36.899171 21668 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:36.902251 24272 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:36.902748 24450 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:36.903460 24450 raft_consensus.cc:695] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:36.906078 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.906492 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.907097 21546 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:36.907361 21554 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:36.907863 24376 raft_consensus.cc:3049] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:36.908216 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.909253 21549 raft_consensus.cc:2405] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250114 08:59:36.910202 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.912556 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:36.865106 24175 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:36.914124 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:36.914294 24291 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.198s
I20250114 08:59:36.915104 21695 raft_consensus.cc:2388] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:36.916870 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:36.918587 24366 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.919025 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:36.919773 21696 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:36.919351 24366 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.908262 24376 raft_consensus.cc:738] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:36.921047 24343 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.921252 21696 raft_consensus.cc:1235] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:36.921566 24094 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.902282 24033 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:36.899276 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:36.914505 24175 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.916179 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:36.923396 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 198099us
I20250114 08:59:36.918598 21702 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:36.934829 21695 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:36.876013 24334 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.949271 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:36.949406 24033 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.950596 24352 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:36.951817 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1538) took 1400 ms. Trace:
I20250114 08:59:36.921744 24343 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.919275 24075 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:36.963492 21696 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250114 08:59:36.969671 24071 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:36.970263 24334 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.971333 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1484) took 2196 ms. Trace:
I20250114 08:59:36.970307 24071 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.973641 24325 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250114 08:59:36.968066 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 235362us
I20250114 08:59:36.975401 24071 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.921377 24376 consensus_queue.cc:260] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:36.965426 24315 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:36.952256 21550 rpcz_store.cc:276] 0114 08:59:35.551679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.705910 (+154231us) service_pool.cc:225] Handling call
0114 08:59:35.743875 (+ 37965us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.744163 (+ 288us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:35.744579 (+ 416us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.295343 (+550764us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b18000bad60. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fe2b 00007fa63f5d797a
0114 08:59:36.922165 (+626822us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:36.939847 (+ 17682us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:36.943865 (+ 4018us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.951499 (+ 7634us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.951657 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":231255,"prepare.queue_time_us":190,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":3975,"raft.queue_time_us":238185,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":8894,"spinlock_wait_cycles":3243776,"thread_start_us":171305,"threads_started":1,"wal-append.queue_time_us":976536}
I20250114 08:59:36.991272 21555 raft_consensus.cc:2463] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 10.
I20250114 08:59:36.939795 24094 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:36.991072 24466 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.003080 24376 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:37.001098 24334 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:36.947795 24450 consensus_queue.cc:237] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.008055 24188 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:37.049067 24188 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:37.053833 24274 raft_consensus.cc:3049] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:37.077471 24033 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:37.096171 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 158850us
I20250114 08:59:37.049396 24466 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:37.089018 24284 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.579s user 0.008s sys 0.008s
I20250114 08:59:37.081785 24188 raft_consensus.cc:3054] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:37.099905 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 111364us
I20250114 08:59:37.123157 24334 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.123785 24319 consensus_queue.cc:1046] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:37.123951 24334 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.124962 24319 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:37.125131 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:37.126044 22272 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:37.125623 24319 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.127000 24446 raft_consensus.cc:3049] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:37.127452 24446 raft_consensus.cc:738] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:37.128763 21669 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:37.128593 24446 consensus_queue.cc:260] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.105142 24071 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.130061 24446 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:37.130179 24071 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.124323 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1505) took 1061 ms. Trace:
I20250114 08:59:37.132660 21626 rpcz_store.cc:276] 0114 08:59:36.062741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.446722 (+383981us) service_pool.cc:225] Handling call
0114 08:59:37.123490 (+676768us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b5800002d98. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:37.123785 (+ 295us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:37.124031 (+ 246us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:37.124055 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:37.124092 (+ 37us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:37.124125 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:37.124173 (+ 48us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:37.124261 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7133440}
W20250114 08:59:37.133249 24500 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20250114 08:59:37.126231 24334 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.083123 24366 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:37.145403 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 62343.8us
I20250114 08:59:37.146814 21521 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:37.147761 24446 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:37.148272 24446 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:37.149081 24446 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:37.150766 24504 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20250114 08:59:37.152405 24498 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
W20250114 08:59:37.153034 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 163599us
I20250114 08:59:37.155820 24502 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:37.156512 24502 raft_consensus.cc:695] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:37.157534 24502 consensus_queue.cc:237] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.093207 24274 raft_consensus.cc:738] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:37.115756 22240 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:37.104064 24175 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:37.177858 24396 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.018s
I20250114 08:59:37.179330 24389 consensus_queue.cc:1035] T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:37.181628 24199 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:37.183331 24343 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.184911 24446 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.187996 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 204) took 1614 ms. Trace:
I20250114 08:59:37.190035 24466 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.190908 24272 consensus_queue.cc:1046] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 08:59:37.191373 21669 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:37.209375 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 210198us
I20250114 08:59:37.177745 24399 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:37.214311 24291 raft_consensus.cc:2743] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:37.215106 24496 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.215488 24184 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20250114 08:59:37.219763 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 97367.1us
I20250114 08:59:37.240783 21594 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:37.112610 24284 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.207073 24094 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:37.221547 21551 rpcz_store.cc:276] 0114 08:59:35.573045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.016980 (+443935us) service_pool.cc:225] Handling call
0114 08:59:36.017320 (+ 340us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:36.019334 (+ 2014us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:36.019364 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:36.019810 (+ 446us) log.cc:844] Serialized 26 byte log entry
0114 08:59:36.484040 (+464230us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:36.484104 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.484233 (+ 129us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:37.187590 (+703357us) raft_consensus.cc:1680] finished
0114 08:59:37.187617 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:37.187778 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5229,"raft.queue_time_us":1664,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4497,"thread_start_us":438740,"threads_started":1,"wal-append.queue_time_us":928372}
I20250114 08:59:37.220994 24398 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:37.224892 24071 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.225878 24353 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.226105 24420 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:37.225688 24175 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:37.238169 24319 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:37.217972 24296 consensus_queue.cc:1035] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20250114 08:59:37.270450 21698 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:37.272173 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.168136 24453 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20250114 08:59:37.236479 24366 consensus_queue.cc:1046] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.040s
I20250114 08:59:37.306313 24366 consensus_queue.cc:1225] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer cec0ef3bbb654988bf2123d8eef7a0a7 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20250114 08:59:37.288187 24395 consensus_meta.cc:287] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.794s user 0.012s sys 0.003s
I20250114 08:59:37.288008 24296 consensus_queue.cc:1035] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250114 08:59:37.288219 24284 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:37.279237 24175 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.317775 24175 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:37.270454 24199 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.320075 24199 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:37.279912 24502 raft_consensus.cc:2798] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:37.323864 24502 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:37.291546 23801 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:37.293432 24319 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:37.293548 24094 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.260615 24325 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:37.295508 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.296653 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:37.300750 24336 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.300073 24274 consensus_queue.cc:260] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.301270 24030 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:37.304620 24398 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.261587 24496 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.290761 24424 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.294776 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.311456 21702 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:37.324314 24502 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:37.347628 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:37.355881 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 124033us
I20250114 08:59:37.270264 24399 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:37.417119 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 84390.2us
W20250114 08:59:37.413944 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57749.7us
I20250114 08:59:37.424661 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:37.428771 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.416850 24094 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.331223 23801 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.312423 24420 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.417600 24274 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:37.331615 24353 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.424175 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:37.459707 21551 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
W20250114 08:59:37.480510 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 289164us
I20250114 08:59:37.491276 24496 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.491894 24366 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.492537 24366 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.495258 24480 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.495936 24480 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.476387 24446 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.497051 24446 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:37.498057 24480 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.500207 23801 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.500757 24420 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.500958 22267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 435) took 1107 ms. Trace:
I20250114 08:59:37.497743 24446 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.503504 24446 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.505916 24353 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.500772 21550 raft_consensus.cc:1270] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:37.485234 21521 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:37.495443 24075 consensus_meta.cc:287] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.000s
I20250114 08:59:37.501420 22267 rpcz_store.cc:276] 0114 08:59:36.393601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.406711 (+ 13110us) service_pool.cc:225] Handling call
0114 08:59:37.405331 (+998620us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656d0e 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:37.500875 (+ 95544us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77287168}
I20250114 08:59:37.507130 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.525357 21702 raft_consensus.cc:1808] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:37.555409 21702 raft_consensus.cc:2405] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:37.421667 24325 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:37.566231 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 81091.3us
I20250114 08:59:37.414108 24030 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.575228 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:37.575418 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:37.581593 24346 consensus_meta.cc:287] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.762s user 0.015s sys 0.000s
I20250114 08:59:37.582628 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:37.587905 22271 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:37.427379 24336 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.439052 24424 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.425659 24319 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.595733 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.598960 24398 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:37.599186 21546 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:37.593361 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.600018 21699 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:37.600293 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:37.600860 24505 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
I20250114 08:59:37.601217 21546 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:37.601260 24336 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.601260 21699 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:37.602298 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.602134 21699 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:37.603682 21546 raft_consensus.cc:1235] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:37.603402 21699 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.605020 21699 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:37.605206 21546 raft_consensus.cc:1270] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:37.605537 24424 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.593099 21553 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:37.608613 24395 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.588876 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.609294 24395 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:37.610641 22271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 436) took 1216 ms. Trace:
I20250114 08:59:37.595494 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.593330 24453 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250114 08:59:37.593225 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:37.611045 22271 rpcz_store.cc:276] 0114 08:59:36.394223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.406711 (+ 12488us) service_pool.cc:225] Handling call
0114 08:59:37.481964 (+1075253us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656d0e 0000000000353db9 00007fa6444b6a50 00007fa64457c814 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:37.610575 (+128611us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58491520}
I20250114 08:59:37.627563 21554 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:37.620189 24366 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:37.628911 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:37.629407 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:37.629906 22274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 425) took 2803 ms (client timeout 1929 ms). Trace:
I20250114 08:59:37.593552 24325 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:37.630561 21553 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
W20250114 08:59:37.630384 22274 rpcz_store.cc:269] 0114 08:59:34.826646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:34.833039 (+ 6393us) service_pool.cc:225] Handling call
0114 08:59:36.407485 (+1574446us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:37.625098 (+1217613us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656cbc 00007fa63c65687b 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:37.629811 (+ 4713us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13089408}
I20250114 08:59:37.590929 24030 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.637902 24188 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.640640 24188 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:37.648136 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 123184us
W20250114 08:59:37.649127 21623 consensus_meta.cc:287] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.714s user 0.008s sys 0.004s
I20250114 08:59:37.649607 21623 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:37.646564 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1503) took 1125 ms. Trace:
I20250114 08:59:37.655638 22239 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:37.659765 21555 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:37.664680 21699 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20250114 08:59:37.671788 21699 pending_rounds.cc:77] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting all ops after (but not including) 3
I20250114 08:59:37.661778 21669 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:37.673990 24353 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:37.672140 21699 pending_rounds.cc:99] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7: Aborting uncommitted NO_OP operation due to leader change: 7.4
W20250114 08:59:37.720372 24520 log.cc:926] Time spent T 82151d61cef04050b69f24bc614507fc P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
W20250114 08:59:37.725307 24523 log.cc:926] Time spent T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
I20250114 08:59:37.661553 24030 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:37.666327 24496 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:37.726872 24496 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:37.727414 24496 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:37.731688 24352 consensus_meta.cc:287] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.780s user 0.012s sys 0.001s
I20250114 08:59:37.726301 24030 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:37.681541 21699 raft_consensus.cc:2883] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:37.661921 24325 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20250114 08:59:36.971738 21622 rpcz_store.cc:276] 0114 08:59:34.774638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.800115 (+1025477us) service_pool.cc:225] Handling call
0114 08:59:35.812441 (+ 12326us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:35.820370 (+ 7929us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:36.076080 (+255710us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.898606 (+822526us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.899064 (+ 458us) log.cc:844] Serialized 14 byte log entry
0114 08:59:36.950926 (+ 51862us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:36.951001 (+ 75us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:36.960368 (+ 9367us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:36.966851 (+ 6483us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:36.971269 (+ 4418us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91133,"prepare.queue_time_us":246891,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":89,"raft.queue_time_us":114718,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":6411,"thread_start_us":879552,"threads_started":2,"wal-append.queue_time_us":729795}
I20250114 08:59:37.963232 21549 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:37.869658 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.764108 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.652285 21703 rpcz_store.cc:276] 0114 08:59:36.516793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.529103 (+ 12310us) service_pool.cc:225] Handling call
0114 08:59:36.529613 (+ 510us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:36.530262 (+ 649us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:36.530296 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:36.939253 (+408957us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b2c00042708. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63dfae741 00007fa63dfbdd72 00007fa63f609141 00007fa63f5cd58a 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782
0114 08:59:36.977646 (+ 38393us) log.cc:844] Serialized 26 byte log entry
0114 08:59:37.226610 (+248964us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:37.250573 (+ 23963us) log.cc:844] Serialized 14 byte log entry
0114 08:59:37.250732 (+ 159us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:37.250791 (+ 59us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:37.631114 (+380323us) raft_consensus.cc:1680] finished
0114 08:59:37.641606 (+ 10492us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:37.641750 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13232,"prepare.queue_time_us":23828,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5274,"raft.queue_time_us":74300,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":46482,"spinlock_wait_cycles":53069056,"thread_start_us":219357,"threads_started":1,"wal-append.queue_time_us":496236}
W20250114 08:59:37.966480 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 171810us
I20250114 08:59:37.973645 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:37.974654 21554 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 9: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:37.974783 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.975945 21554 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:37.976663 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.978175 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.985211 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:37.985913 21555 raft_consensus.cc:1808] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Handling vote request from an unknown peer cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:37.986410 21555 raft_consensus.cc:2388] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 6: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:37.905213 24376 consensus_meta.cc:287] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.862s user 0.013s sys 0.000s
I20250114 08:59:37.734831 24315 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:37.986493 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.992645 21701 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:37.996488 24030 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.962066 24319 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:37.997673 24376 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:37.998617 24376 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:37.992147 24274 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.531s user 0.010s sys 0.001s
I20250114 08:59:38.004787 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.006289 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.007503 21702 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:38.008750 24397 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.009300 24397 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.008792 24424 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.010776 24424 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.011547 24397 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:37.828641 24527 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:38.012356 24527 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.012791 24527 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:38.013442 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.015075 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.016930 21695 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:38.004839 24395 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:38.018014 21521 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:38.020824 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.021775 21699 raft_consensus.cc:2405] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250114 08:59:37.795790 24399 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.029728 24325 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:38.030346 24325 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:38.032296 24271 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.021068 24315 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.032886 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:37.794920 24375 consensus_queue.cc:1035] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.228s
I20250114 08:59:38.038220 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.045058 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:38.045811 21553 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:38.053562 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:38.025377 24319 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.058331 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.032430 24325 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:38.032850 24271 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.060563 24319 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.060933 24424 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.091269 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.091519 24272 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:38.101974 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:38.092090 24502 consensus_meta.cc:287] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.666s user 0.005s sys 0.006s
I20250114 08:59:38.093086 24199 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.095276 24370 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.096089 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:38.096858 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:38.062060 24395 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:38.097513 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.097970 21625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1710) took 1383 ms. Trace:
I20250114 08:59:38.106786 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:38.104269 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:38.110949 21629 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:38.111552 21621 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:38.102938 21521 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:38.106406 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:38.098757 24334 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:38.106431 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 865702us
I20250114 08:59:38.106184 24502 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.115082 21594 leader_election.cc:304] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:38.108417 21623 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:38.110126 21626 raft_consensus.cc:1270] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250114 08:59:38.113225 24322 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.113334 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.113606 24071 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:38.098484 24025 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.117449 21704 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:38.102923 24315 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.125020 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.125994 21630 raft_consensus.cc:1235] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:38.122455 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.110462 21625 rpcz_store.cc:276] 0114 08:59:36.714531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.721370 (+ 6839us) service_pool.cc:225] Handling call
0114 08:59:36.736212 (+ 14842us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:36.846988 (+110776us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:36.847027 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:36.896692 (+ 49665us) log.cc:844] Serialized 26 byte log entry
0114 08:59:37.079287 (+182595us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:37.164568 (+ 85281us) log.cc:844] Serialized 14 byte log entry
0114 08:59:38.097666 (+933098us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:38.097731 (+ 65us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:38.097765 (+ 34us) raft_consensus.cc:1680] finished
0114 08:59:38.097777 (+ 12us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:38.097894 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1103768,"prepare.queue_time_us":34953,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":107156,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":5049,"thread_start_us":118389,"threads_started":1,"wal-append.queue_time_us":251296}
I20250114 08:59:38.119717 24334 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.126904 24525 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:38.126938 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 918011us
I20250114 08:59:38.107630 24376 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:38.110107 24395 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.127748 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.129333 21701 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 8: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:38.099543 24496 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:38.160666 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:38.197607 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 67151.1us
I20250114 08:59:38.120743 24071 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.203620 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.124305 24424 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:38.116237 24468 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:38.205235 21629 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
W20250114 08:59:38.205652 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 90662.1us
I20250114 08:59:38.207485 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.210013 24496 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.210141 21521 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:38.128088 24030 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20250114 08:59:38.232321 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.128625 24399 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:38.221350 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:38.206374 24395 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:38.127338 24025 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.214381 24071 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:38.122241 24199 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.118490 24370 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:38.124065 24272 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:38.233274 24480 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:38.270439 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.205744 24525 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:38.129768 24322 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.278007 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.292150 21699 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:38.298381 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.299677 21521 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:38.254154 24375 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:38.220654 24468 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.307313 21704 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:38.308290 21622 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:38.291235 24480 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.319904 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.299985 24376 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.321420 21699 raft_consensus.cc:2405] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250114 08:59:38.327445 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.328584 21699 raft_consensus.cc:2405] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:38.329668 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.306528 24075 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:38.333117 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 184925us
I20250114 08:59:38.316284 24366 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.322908 21596 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:38.336364 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 117192us
W20250114 08:59:38.334985 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 81082.4us
I20250114 08:59:38.324710 24480 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:38.297343 24399 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:38.334352 24537 log.cc:926] Time spent T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20250114 08:59:38.341516 21699 raft_consensus.cc:1212] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Deduplicated request from leader. Original: 4.3->[6.4-8.5] Dedup: 8.5->[]
I20250114 08:59:38.345456 24181 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.373131 24025 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.373459 21699 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20250114 08:59:38.373795 24025 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.374342 24025 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.375437 24480 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:38.373481 24181 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.377590 24480 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:38.346426 24532 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.223s user 0.000s sys 0.001s
I20250114 08:59:38.350203 24399 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.350761 24502 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.328513 24376 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.352674 24370 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:38.345798 24310 raft_consensus.cc:2798] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:38.335951 24271 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.407802 21594 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:38.355988 24188 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.335747 24319 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.356534 24366 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.339650 24075 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.358158 24199 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.358264 21697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46402 (request call id 1502) took 1841 ms. Trace:
I20250114 08:59:38.310158 24375 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.363641 24334 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.377910 24181 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.356326 24527 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.345139 24030 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.345772 24530 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:38.345173 21593 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:38.381312 21703 raft_consensus.cc:2463] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
W20250114 08:59:38.388660 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 203135us
I20250114 08:59:38.399405 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:38.376358 24025 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.406594 24310 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.580037 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 208) took 1262 ms. Trace:
I20250114 08:59:38.437451 24370 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250114 08:59:38.580981 21548 rpcz_store.cc:276] 0114 08:59:37.317414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:37.317557 (+ 143us) service_pool.cc:225] Handling call
0114 08:59:37.317965 (+ 408us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:37.318409 (+ 444us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:37.318435 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:37.318994 (+ 559us) log.cc:844] Serialized 26 byte log entry
0114 08:59:37.444794 (+125800us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:37.444860 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:37.444959 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:38.579736 (+1134777us) raft_consensus.cc:1680] finished
0114 08:59:38.579766 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:38.579957 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4480,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":125354,"threads_started":1,"wal-append.queue_time_us":742729}
I20250114 08:59:38.510784 21697 rpcz_store.cc:276] 0114 08:59:36.516280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.527834 (+ 11554us) service_pool.cc:225] Handling call
0114 08:59:36.528373 (+ 539us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:36.624907 (+ 96534us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:36.624944 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:36.625609 (+ 665us) log.cc:844] Serialized 26 byte log entry
0114 08:59:37.928579 (+1302970us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b18003a83a0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:59:38.042985 (+114406us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:38.043027 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:38.043092 (+ 65us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:38.357736 (+314644us) raft_consensus.cc:1680] finished
0114 08:59:38.357764 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:38.358057 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96562,"raft.queue_time_us":96738,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":156,"spinlock_wait_cycles":3997056,"thread_start_us":114198,"threads_started":1,"wal-append.queue_time_us":1481694}
I20250114 08:59:38.397835 24336 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
I20250114 08:59:38.590809 21702 raft_consensus.cc:2363] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:38.436723 24480 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:38.595063 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 295445us
I20250114 08:59:38.601729 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.552577 24530 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.585196 24310 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:38.438216 24188 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:38.632565 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 253047us
W20250114 08:59:38.635396 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 312524us
I20250114 08:59:38.642592 21596 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:38.646255 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 302003us
I20250114 08:59:38.647300 21593 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:38.662644 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:38.604393 24025 raft_consensus.cc:2743] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:38.664213 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:38.670648 21703 raft_consensus.cc:3054] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 7
I20250114 08:59:38.578994 24319 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250114 08:59:38.668861 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 299211us
I20250114 08:59:38.587450 24030 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.667810 21520 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:38.604209 24334 raft_consensus.cc:2798] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250114 08:59:38.625135 24373 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:38.659132 24468 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.652894 24525 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.663647 24322 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.677112 24319 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:38.689507 21630 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:38.639261 24480 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.675617 24530 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:38.675261 24551 raft_consensus.cc:491] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:38.684957 24334 raft_consensus.cc:491] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.684985 24547 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:38.763707 24025 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:38.708995 24545 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:38.747493 24319 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:38.788776 22270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51560 (request call id 26) took 1113 ms. Trace:
I20250114 08:59:38.789219 22270 rpcz_store.cc:276] 0114 08:59:37.675280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.787885 (+1112605us) service_pool.cc:225] Handling call
0114 08:59:38.788697 (+ 812us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:38.790855 21552 raft_consensus.cc:3049] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:38.791251 21552 raft_consensus.cc:738] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:38.792021 21552 consensus_queue.cc:260] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.744678 24468 raft_consensus.cc:491] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.793102 21552 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:38.793380 24468 raft_consensus.cc:513] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.796007 24468 leader_election.cc:290] T 270ce1b254bd4b7180e48e18af79cda6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:38.797616 21551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 207) took 1489 ms (client timeout 1843 ms). Trace:
W20250114 08:59:38.798032 21551 rpcz_store.cc:269] 0114 08:59:37.308313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:37.308590 (+ 277us) service_pool.cc:225] Handling call
0114 08:59:38.797415 (+1488825us) spinlock_profiling.cc:243] Waited 54.4 ms on lock 0x7b54001171d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:38.797555 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125049216}
I20250114 08:59:38.798693 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:38.799382 21697 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:38.800197 21550 raft_consensus.cc:1270] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250114 08:59:38.801685 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1559) took 1205 ms. Trace:
I20250114 08:59:38.802157 24181 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.802098 21550 rpcz_store.cc:276] 0114 08:59:37.611572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:37.627291 (+ 15719us) service_pool.cc:225] Handling call
0114 08:59:38.799855 (+1172564us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:38.800629 (+ 774us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:38.801500 (+ 871us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001171d0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:38.801630 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2692224768}
I20250114 08:59:38.802806 24181 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.804852 24075 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.776669 24094 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:38.777004 24540 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20250114 08:59:38.777119 24545 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:38.806249 24554 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:38.776067 24542 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20250114 08:59:38.771804 24025 raft_consensus.cc:491] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:38.778226 24543 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:38.778559 24544 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20250114 08:59:38.779426 24399 consensus_queue.cc:1035] T d0b4ce684a6943809542800e14d94079 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:59:38.779999 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:38.781144 24375 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.782068 21698 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1962) took 1872 ms. Trace:
I20250114 08:59:38.747673 24373 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.785876 24071 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.804555 24547 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:38.777789 24541 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250114 08:59:38.748808 24550 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.801581 21552 raft_consensus.cc:1270] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250114 08:59:38.782950 24552 raft_consensus.cc:2743] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:38.828886 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:38.957816 21698 rpcz_store.cc:276] 0114 08:59:36.909300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.912563 (+ 3263us) service_pool.cc:225] Handling call
0114 08:59:36.912920 (+ 357us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:37.275616 (+362696us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:38.781801 (+1506185us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400371ed4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:38.781998 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":821796096}
I20250114 08:59:38.928725 21668 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:38.758863 24334 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:38.839885 24468 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:38.971959 24375 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:38.895021 24366 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:38.953934 24536 log.cc:926] Time spent T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20250114 08:59:38.812053 21703 raft_consensus.cc:2463] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:38.973910 24550 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:38.806496 24030 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:38.929005 24376 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:38.968997 24025 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:38.866104 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:38.770476 24553 raft_consensus.cc:491] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:38.972330 24527 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:39.014704 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 346979us
I20250114 08:59:39.029639 21695 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 57f9cc1dd3c64647848a33a65b08c893 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:39.031221 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 384013us
I20250114 08:59:39.051348 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:39.053411 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 371771us
W20250114 08:59:39.049821 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 54319.5us
I20250114 08:59:39.090023 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:38.989563 24188 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:39.054281 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 411782us
I20250114 08:59:39.092054 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.092564 21596 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:39.093786 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 265566us
I20250114 08:59:38.761277 24551 raft_consensus.cc:513] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.104436 21553 raft_consensus.cc:1270] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250114 08:59:39.107806 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.035349 24181 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.114557 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.112880 21548 raft_consensus.cc:2463] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:39.123049 21551 raft_consensus.cc:2405] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250114 08:59:39.122795 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:39.127832 21553 raft_consensus.cc:2388] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:38.981206 24543 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:39.130038 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 301214us
I20250114 08:59:39.130734 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1719) took 1034 ms. Trace:
I20250114 08:59:39.148713 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.149958 21546 raft_consensus.cc:2405] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 08:59:39.105764 24527 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:38.980350 24554 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.143952 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 209) took 1684 ms. Trace:
I20250114 08:59:39.110575 24468 raft_consensus.cc:695] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:39.122749 24188 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:39.169793 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.124559 24569 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.171981 21521 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.157313 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.185211 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.127768 24480 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.163017 21547 rpcz_store.cc:276] 0114 08:59:37.459984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:37.460182 (+ 198us) service_pool.cc:225] Handling call
0114 08:59:37.648830 (+188648us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:37.977890 (+329060us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:37.977939 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:37.991677 (+ 13738us) log.cc:844] Serialized 26 byte log entry
0114 08:59:38.336523 (+344846us) raft_consensus.cc:1640] Marking committed up to 8
0114 08:59:38.336572 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:38.336637 (+ 65us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:39.143685 (+807048us) raft_consensus.cc:1680] finished
0114 08:59:39.143703 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:39.143877 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35820,"raft.queue_time_us":286872,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":1775,"thread_start_us":287041,"threads_started":1,"wal-append.queue_time_us":552637}
I20250114 08:59:39.190919 21625 raft_consensus.cc:1808] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Handling vote request from an unknown peer cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:39.141916 24375 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.142073 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:39.126849 24315 consensus_queue.cc:1035] T a03f2f2f70724b85b8b23ed7bc72fbe1 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20250114 08:59:39.051113 24553 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.160845 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:39.198588 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:39.326217 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.147683 21628 rpcz_store.cc:276] 0114 08:59:38.096244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.096371 (+ 127us) service_pool.cc:225] Handling call
0114 08:59:38.096761 (+ 390us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:38.097139 (+ 378us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:38.097166 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:38.101202 (+ 4036us) log.cc:844] Serialized 26 byte log entry
0114 08:59:38.334685 (+233483us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:38.670462 (+335777us) log.cc:844] Serialized 14 byte log entry
0114 08:59:38.754222 (+ 83760us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:38.754318 (+ 96us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:39.130488 (+376170us) raft_consensus.cc:1680] finished
0114 08:59:39.130522 (+ 34us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:39.130667 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":100761,"prepare.queue_time_us":430212,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":134145,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":5688,"thread_start_us":540811,"threads_started":2,"wal-append.queue_time_us":330214}
I20250114 08:59:39.404666 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.237119 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.414113 21553 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 5.
I20250114 08:59:39.393545 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.394098 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.341711 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.199815 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:39.417729 21626 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:39.418231 21624 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:39.419391 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.415638 21547 raft_consensus.cc:2463] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:39.420754 22272 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
W20250114 08:59:39.434871 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 328457us
I20250114 08:59:39.420363 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:39.436120 21624 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:39.198731 21625 raft_consensus.cc:2405] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20250114 08:59:39.421200 21623 raft_consensus.cc:2363] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
I20250114 08:59:39.434578 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.438666 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.414361 21546 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:39.416924 21551 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
W20250114 08:59:39.238483 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 66629.3us
W20250114 08:59:39.393224 24310 consensus_meta.cc:287] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.729s user 0.009s sys 0.001s
W20250114 08:59:39.323042 24544 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20250114 08:59:39.459805 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.438855 21621 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:39.396281 24524 log.cc:926] Time spent T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 1.749s user 0.000s sys 0.019s
I20250114 08:59:39.170385 24545 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.190521 24181 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:39.396111 24566 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:39.466076 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:39.396747 24564 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.467765 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:39.477623 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 61842.6us
I20250114 08:59:39.169346 24527 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.191054 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:39.454422 24550 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:39.453979 24525 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.000s sys 0.012s
W20250114 08:59:39.465034 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 372572us
I20250114 08:59:39.471625 21623 raft_consensus.cc:1270] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 08:59:39.472954 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.471623 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.496868 21697 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:39.495965 21695 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:39.318512 24573 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
W20250114 08:59:39.438997 21520 connection.cc:232] RPC call timeout handler was delayed by -1.1024835639999537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.167s for a response
I20250114 08:59:39.358212 24373 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.284018 24071 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.320972 24547 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:39.478438 21668 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:39.284260 24480 raft_consensus.cc:2798] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:39.470505 21625 raft_consensus.cc:1270] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:39.477823 21703 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:39.493834 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.212332 24030 consensus_queue.cc:1046] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.264s
I20250114 08:59:39.454456 22266 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:39.495328 21546 raft_consensus.cc:1235] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:39.494783 24525 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.610s user 0.000s sys 0.012s
I20250114 08:59:39.499748 24545 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:39.500275 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.523242 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:39.191102 24188 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.462621 24376 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20250114 08:59:39.211699 24569 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.506428 21668 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:39.492127 24366 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:39.500599 24533 log.cc:926] Time spent T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20250114 08:59:39.542680 24545 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.544911 24366 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.543061 24502 raft_consensus.cc:2758] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20250114 08:59:39.546265 24319 consensus_meta.cc:287] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.757s user 0.011s sys 0.000s
I20250114 08:59:39.546599 24502 consensus_queue.cc:1035] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250114 08:59:39.547009 24545 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.546708 24319 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.547187 24366 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.498055 22268 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:39.465549 24310 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.548760 24580 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250114 08:59:39.549486 24395 consensus_meta.cc:287] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 1.272s user 0.014s sys 0.000s
I20250114 08:59:39.549913 24395 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.550844 24353 raft_consensus.cc:491] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:39.551123 21553 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250114 08:59:39.551278 24353 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:39.517268 24094 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.519852 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.521138 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.522466 24030 raft_consensus.cc:3049] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:39.523080 24554 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.458045 24468 consensus_queue.cc:237] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.531168 24571 raft_consensus.cc:491] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:39.508078 24550 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.530557 21549 raft_consensus.cc:1270] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:39.438076 24375 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:39.507115 21593 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.132:35045 timed out after 1.988s (SENT) [suppressed 12 similar messages]
I20250114 08:59:39.582865 21596 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:39.531288 24594 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:39.550222 24310 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.542644 24315 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:39.519017 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.498442 24566 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:39.531098 24568 log.cc:926] Time spent T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
I20250114 08:59:39.548761 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:39.552846 21550 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:39.525049 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.531627 24480 raft_consensus.cc:695] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:39.544078 21521 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.510799 24551 leader_election.cc:290] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.531807 21695 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250114 08:59:39.541692 24547 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.508078 24399 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:39.548188 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.630964 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.490170 24564 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.530968 24549 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:39.540426 24525 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:39.634769 24525 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.550266 21449 catalog_manager.cc:5526] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:39.637145 24525 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.637763 24525 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.633936 22241 leader_election.cc:304] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:39.637966 21622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1711) took 1544 ms (client timeout 1954 ms). Trace:
W20250114 08:59:39.638381 24025 consensus_meta.cc:287] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.584s user 0.009s sys 0.000s
I20250114 08:59:39.639143 24366 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:39.645644 21623 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
W20250114 08:59:39.631791 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 87821.8us
I20250114 08:59:39.633414 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.647341 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.648900 21546 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 10.
I20250114 08:59:39.649279 21548 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:39.631711 21549 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
W20250114 08:59:39.641206 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 183841us
W20250114 08:59:39.640632 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 57853.1us
I20250114 08:59:39.507436 24573 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:39.646112 21622 rpcz_store.cc:269] 0114 08:59:38.093726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.095758 (+ 2032us) service_pool.cc:225] Handling call
0114 08:59:39.637739 (+1541981us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54002909d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:39.637870 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459817728}
I20250114 08:59:39.643484 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:39.654093 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 176570us
I20250114 08:59:39.577905 24030 raft_consensus.cc:738] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:39.652668 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:39.655656 21555 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:39.625242 24336 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.638522 21449 catalog_manager.cc:5526] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 6 to 7, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 7 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:39.657485 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.644788 24395 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.629809 24424 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:39.665900 21521 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.665592 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.629638 24373 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:39.606437 24550 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.672611 21630 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:39.704962 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 163321us
I20250114 08:59:39.705886 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:39.607517 24094 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.651048 24366 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:39.717106 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:39.716336 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 202652us
W20250114 08:59:39.672789 24530 consensus_meta.cc:287] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.909s user 0.011s sys 0.002s
I20250114 08:59:39.665591 24590 raft_consensus.cc:2798] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:39.724927 21622 raft_consensus.cc:2388] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:39.755563 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:39.757314 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 101121us
I20250114 08:59:39.629796 24315 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.608147 24594 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.769768 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.770574 21695 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:39.771507 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.771999 24594 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:39.772215 21698 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
I20250114 08:59:39.772980 24588 raft_consensus.cc:2743] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 08:59:39.773186 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.773820 24375 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:39.773972 21697 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
W20250114 08:59:39.769204 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 103375us
I20250114 08:59:39.778712 21630 raft_consensus.cc:2388] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:39.645112 24547 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.755676 21593 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:39.652211 24025 raft_consensus.cc:513] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:39.783195 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 173166us
W20250114 08:59:39.781683 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 121028us
I20250114 08:59:39.651504 24525 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.725332 24366 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:39.792008 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.794515 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.719133 24595 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:39.672426 24336 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.756603 24527 leader_election.cc:290] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:39.650801 24549 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.708662 24424 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.709226 24373 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.785246 24590 raft_consensus.cc:695] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:39.789826 24553 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.851867 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.852982 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:39.857657 21622 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:39.651569 24571 raft_consensus.cc:513] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.862058 21546 raft_consensus.cc:1270] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:39.863600 21667 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.864537 24566 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.865448 24569 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.866191 24556 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250114 08:59:39.866729 24556 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:39.867324 24556 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:39.756629 24530 raft_consensus.cc:513] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.801757 24595 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:39.850050 24593 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:39.854763 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.872535 21547 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:39.872718 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 213342us
I20250114 08:59:39.875384 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:39.882709 21668 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.883584 21668 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.884721 21668 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:39.885905 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:39.886205 24550 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.886305 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43436 (request call id 306) took 1261 ms. Trace:
I20250114 08:59:39.885840 24590 consensus_queue.cc:237] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.886670 21695 raft_consensus.cc:3054] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 7
I20250114 08:59:39.886780 21629 rpcz_store.cc:276] 0114 08:59:38.625288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.640680 (+ 15392us) service_pool.cc:225] Handling call
0114 08:59:38.665570 (+ 24890us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:38.682030 (+ 16460us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:38.682062 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:38.682105 (+ 43us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:38.682145 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:39.886084 (+1203939us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b540022db54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:39.886135 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:39.886246 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37341056}
I20250114 08:59:39.888208 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.889300 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.890017 21630 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:39.894109 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:39.894541 24373 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:39.891690 24376 raft_consensus.cc:2798] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250114 08:59:39.895172 24376 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:39.895547 24376 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:39.895488 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:39.896548 24359 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:39.866609 24564 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.897647 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1716) took 1802 ms. Trace:
I20250114 08:59:39.898059 21627 rpcz_store.cc:276] 0114 08:59:38.095296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.095445 (+ 149us) service_pool.cc:225] Handling call
0114 08:59:38.095866 (+ 421us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 08:59:38.406056 (+310190us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:38.406105 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 08:59:38.441457 (+ 35352us) log.cc:844] Serialized 78 byte log entry
0114 08:59:38.637967 (+196510us) raft_consensus.cc:1640] Marking committed up to 2
0114 08:59:39.090451 (+452484us) log.cc:844] Serialized 14 byte log entry
0114 08:59:39.453093 (+362642us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:39.897210 (+444117us) spinlock_profiling.cc:243] Waited 9.59 ms on lock 0x7b54001527d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:39.897261 (+ 51us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:39.897312 (+ 51us) raft_consensus.cc:1680] finished
0114 08:59:39.897415 (+ 103us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:39.897587 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107305,"prepare.queue_time_us":333891,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":343164,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":38540,"spinlock_wait_cycles":22050176,"thread_start_us":1021666,"threads_started":4,"wal-append.queue_time_us":267981}
I20250114 08:59:39.901158 24336 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:39.901716 24334 consensus_meta.cc:287] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.861s user 0.006s sys 0.004s
I20250114 08:59:39.902139 24334 raft_consensus.cc:513] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:39.902505 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.903754 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:39.904024 24334 leader_election.cc:290] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:39.905279 24589 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.254s user 0.001s sys 0.000s
I20250114 08:59:39.780792 24480 consensus_queue.cc:237] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:39.906090 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.857241 24188 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.894289 24386 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:39.907840 21623 raft_consensus.cc:2363] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
I20250114 08:59:39.908147 21629 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:39.900261 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:39.919118 21626 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:39.924587 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:39.971390 21624 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.514s user 0.013s sys 0.003s
I20250114 08:59:39.970489 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:39.972896 21630 raft_consensus.cc:3054] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:39.798969 24030 consensus_queue.cc:260] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.215461 22270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:40.110934 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 392860us
I20250114 08:59:39.879484 24595 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:39.996959 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 71272.6us
I20250114 08:59:40.225167 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.225729 21623 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:40.035138 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.879472 24375 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:39.971750 21624 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:40.091784 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:40.232599 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 342079us
W20250114 08:59:40.230682 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 345990us
I20250114 08:59:39.973829 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:39.887583 24094 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.188202 21695 raft_consensus.cc:2463] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:39.892275 24588 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:39.970464 21627 raft_consensus.cc:2371] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9. Re-sending same reply.
I20250114 08:59:40.255259 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.259295 21624 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:40.259533 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.260565 21549 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:40.260690 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.265740 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.268769 21546 raft_consensus.cc:2463] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:40.269903 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:40.270603 21546 raft_consensus.cc:3054] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 4 FOLLOWER]: Advancing to term 6
I20250114 08:59:40.268100 21624 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:40.216672 24495 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:40.191284 24594 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:40.275751 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.969264 24386 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:40.277029 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.281417 21624 raft_consensus.cc:2388] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 11: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:40.223444 24573 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:39.856600 21449 catalog_manager.cc:5526] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 5 to 6, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 6 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:40.244529 24564 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:40.288990 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:39.997031 24450 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:40.289366 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:40.288154 21626 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:40.249320 24025 leader_election.cc:290] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.291709 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.246704 24375 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:40.294543 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:40.325191 21703 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:40.295245 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.295722 24564 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.297638 21449 catalog_manager.cc:5526] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 5, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:40.299132 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.258360 24588 raft_consensus.cc:491] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:40.301019 21696 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1978) took 3004 ms. Trace:
I20250114 08:59:40.304555 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:40.273324 24547 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.274097 24315 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:40.347805 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.348559 21624 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 5.
I20250114 08:59:40.286535 24594 raft_consensus.cc:491] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:40.283324 24495 raft_consensus.cc:491] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:40.290457 24386 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:40.325353 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.327142 21627 raft_consensus.cc:2388] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:40.326188 21623 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:40.352864 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.353745 21549 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:40.330448 21594 leader_election.cc:304] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:40.292268 24525 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:40.300356 24272 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:40.301374 24549 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:40.345583 21621 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.008s sys 0.026s
I20250114 08:59:40.329842 21521 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:40.367697 21621 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.008s sys 0.026s
I20250114 08:59:40.368788 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.344450 21696 rpcz_store.cc:276] 0114 08:59:37.296768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:37.309261 (+ 12493us) service_pool.cc:225] Handling call
0114 08:59:37.348214 (+ 38953us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:37.592370 (+244156us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:37.592402 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:37.592482 (+ 80us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:37.592551 (+ 69us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.300730 (+2708179us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400189cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:40.300794 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:40.300930 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":560884096}
I20250114 08:59:40.301143 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:40.345253 24588 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:40.351548 21622 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:40.354980 24279 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:40.361588 21704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1994) took 2326 ms (client timeout 2202 ms). Trace:
I20250114 08:59:40.384075 21623 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 11 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20250114 08:59:39.907567 24597 consensus_queue.cc:1035] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.488s
I20250114 08:59:40.389096 21521 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:40.390797 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1528) took 1063 ms. Trace:
I20250114 08:59:40.363732 21594 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:40.366636 24547 raft_consensus.cc:2743] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:40.322788 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:40.339151 24424 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.353499 24594 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:40.377761 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.305900 24450 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.353716 24495 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:40.331868 24375 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:40.352371 24530 leader_election.cc:290] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:40.383842 21704 rpcz_store.cc:269] 0114 08:59:38.035033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.239218 (+204185us) service_pool.cc:225] Handling call
0114 08:59:40.361305 (+2122087us) spinlock_profiling.cc:243] Waited 9.51 ms on lock 0x7b54002ea254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:40.361450 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21861632}
I20250114 08:59:40.391314 21623 rpcz_store.cc:276] 0114 08:59:39.327042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.364746 (+1037704us) service_pool.cc:225] Handling call
0114 08:59:40.383871 (+ 19125us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:40.390431 (+ 6560us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.390727 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:40.401345 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:40.402415 21594 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:40.387895 24279 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.404147 24025 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:40.404772 24593 consensus_queue.cc:1035] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250114 08:59:40.374540 24606 raft_consensus.cc:2798] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:40.415288 24336 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
W20250114 08:59:40.422093 24399 consensus_meta.cc:287] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.777s user 0.011s sys 0.000s
W20250114 08:59:40.395999 24366 consensus_meta.cc:287] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.578s user 0.005s sys 0.005s
I20250114 08:59:40.400979 24550 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:40.467303 21630 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:40.436172 21698 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Advancing to term 10
W20250114 08:59:40.470414 24556 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.603s user 0.011s sys 0.000s
I20250114 08:59:40.470842 24556 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.471793 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 1966) took 3561 ms. Trace:
I20250114 08:59:40.472139 21700 rpcz_store.cc:276] 0114 08:59:36.910592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.913131 (+ 2539us) service_pool.cc:225] Handling call
0114 08:59:36.913480 (+ 349us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:36.972788 (+ 59308us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 08:59:37.061743 (+ 88955us) log.cc:844] Serialized 14 byte log entry
0114 08:59:37.347998 (+286255us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:37.348165 (+ 167us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:37.348415 (+ 250us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.471472 (+3123057us) spinlock_profiling.cc:243] Waited 52.6 ms on lock 0x7b540023fad4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:40.471515 (+ 43us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:40.471634 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22614,"prepare.queue_time_us":211144,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":49314,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":11899,"spinlock_wait_cycles":121060224,"thread_start_us":321641,"threads_started":3,"wal-append.queue_time_us":214988,"wal-append.run_cpu_time_us":1630,"wal-append.run_wall_time_us":1004747}
W20250114 08:59:40.473757 24376 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.578s user 0.008s sys 0.002s
I20250114 08:59:40.474195 24376 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.476810 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:40.477893 24353 consensus_meta.cc:287] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.926s user 0.012s sys 0.000s
I20250114 08:59:40.478327 21629 raft_consensus.cc:1270] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250114 08:59:40.478297 24353 raft_consensus.cc:513] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.483182 24595 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.484385 22270 raft_consensus.cc:2363] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
I20250114 08:59:40.432014 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:40.485553 21667 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 207744us
I20250114 08:59:40.485754 21695 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
W20250114 08:59:40.487118 22275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49020 (request call id 306) took 2343 ms (client timeout 1694 ms). Trace:
W20250114 08:59:40.487504 22275 rpcz_store.cc:269] 0114 08:59:38.143834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.486936 (+2343102us) service_pool.cc:212] Skipping call since client already timed out
0114 08:59:40.487039 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 08:59:40.459808 24386 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:40.441298 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 551866us
I20250114 08:59:40.442826 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.402144 24094 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:40.495890 24094 raft_consensus.cc:491] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:40.496305 24094 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:40.496613 24571 leader_election.cc:290] T eb00de7db5024821adbfdfd3f8a0f475 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.500133 21626 pending_rounds.cc:77] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting all ops after (but not including) 3
I20250114 08:59:40.500512 21626 pending_rounds.cc:99] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250114 08:59:40.500767 21626 raft_consensus.cc:2883] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 08:59:40.502679 24606 raft_consensus.cc:695] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:40.502625 24526 consensus_queue.cc:1035] T 0564bed845d3430e8831ee6428122144 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:40.503779 24606 consensus_queue.cc:237] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.504388 24279 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:40.450969 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:40.452451 21593 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:40.511641 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 59312.6us
I20250114 08:59:40.457163 24336 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.513549 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1531) took 5443 ms. Trace:
I20250114 08:59:40.460592 22273 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:40.480332 21698 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:40.424026 21449 catalog_manager.cc:5526] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 2 to 5, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:40.461231 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1527) took 1138 ms. Trace:
I20250114 08:59:40.461311 24399 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:40.461802 24366 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.463352 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:40.465898 24450 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:40.465662 24375 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.476351 24376 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:40.467262 24550 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.494124 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:40.578076 21548 raft_consensus.cc:2371] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9. Re-sending same reply.
W20250114 08:59:40.495430 21702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1992) took 2461 ms (client timeout 1851 ms). Trace:
I20250114 08:59:40.505115 24580 raft_consensus.cc:491] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 08:59:40.579327 21702 rpcz_store.cc:269] 0114 08:59:38.034476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.114058 (+ 79582us) service_pool.cc:225] Handling call
0114 08:59:40.495251 (+2381193us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b540023e1d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:40.495368 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":885667584}
I20250114 08:59:40.459550 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:40.514191 22271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49020 (request call id 304) took 2395 ms (client timeout 1835 ms). Trace:
W20250114 08:59:40.585085 22271 rpcz_store.cc:269] 0114 08:59:38.119002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.514013 (+2395011us) service_pool.cc:212] Skipping call since client already timed out
0114 08:59:40.514128 (+ 115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250114 08:59:40.509003 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 53403.8us
I20250114 08:59:40.534441 24319 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:40.437453 21594 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:40.447968 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.513935 21552 rpcz_store.cc:276] 0114 08:59:35.070581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:35.420111 (+349530us) service_pool.cc:225] Handling call
0114 08:59:35.452658 (+ 32547us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:38.790619 (+3337961us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b540010a520. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f55d990 00007fa63f5cf300 00007fa63f5d0371 00007fa63f5ccedf 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc
0114 08:59:38.957451 (+166832us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.513267 (+1555816us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b540007b2d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:40.513463 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4272801024}
I20250114 08:59:40.505964 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:40.536177 24386 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:40.542651 24025 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:40.489049 24588 raft_consensus.cc:513] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.673274 24588 leader_election.cc:290] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.539652 21621 rpcz_store.cc:276] 0114 08:59:39.322418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.469058 (+146640us) service_pool.cc:225] Handling call
0114 08:59:39.480296 (+ 11238us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 08:59:39.748757 (+268461us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:39.748789 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 08:59:39.775389 (+ 26600us) log.cc:844] Serialized 78 byte log entry
0114 08:59:40.368404 (+593015us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:40.368457 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.368577 (+ 120us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:40.461010 (+ 92433us) raft_consensus.cc:1680] finished
0114 08:59:40.461030 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:40.461168 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60612,"raft.queue_time_us":62080,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"thread_start_us":798105,"threads_started":2,"wal-append.queue_time_us":681721}
I20250114 08:59:40.674393 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.466650 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:40.675347 21703 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
W20250114 08:59:40.675609 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 238208us
I20250114 08:59:40.556623 21553 raft_consensus.cc:2363] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9.
I20250114 08:59:40.557567 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:40.558712 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:40.539275 24094 raft_consensus.cc:513] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:40.562520 21555 raft_consensus.cc:2388] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:40.473753 21630 raft_consensus.cc:1270] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 08:59:40.583865 24580 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:40.629593 21521 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:40.637341 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.634225 21546 raft_consensus.cc:2463] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:40.738229 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 108731us
I20250114 08:59:40.608137 21704 raft_consensus.cc:2363] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:40.644173 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.646108 24619 consensus_queue.cc:1035] T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 08:59:40.649771 24543 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.650596 21669 leader_election.cc:400] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
W20250114 08:59:40.656932 22273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49004 (request call id 434) took 4263 ms (client timeout 1988 ms). Trace:
W20250114 08:59:40.659173 21701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 1985) took 2627 ms (client timeout 1756 ms). Trace:
I20250114 08:59:40.666188 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1584) took 1085 ms. Trace:
I20250114 08:59:40.666618 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.630693 24336 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:40.669771 24411 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:40.667440 24564 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.670145 24595 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.670486 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:40.670835 24550 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:40.676854 21594 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:40.734349 22274 raft_consensus.cc:2388] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:40.735687 21630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1540) took 1100 ms. Trace:
I20250114 08:59:40.775970 21630 rpcz_store.cc:276] 0114 08:59:39.635131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.473145 (+838014us) service_pool.cc:225] Handling call
0114 08:59:40.473424 (+ 279us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:40.735324 (+261900us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.735611 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:40.739161 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.777626 21546 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:40.778779 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.739831 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.738194 21548 raft_consensus.cc:2463] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:40.742550 21669 leader_election.cc:403] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20250114 08:59:40.743778 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:40.743973 21449 catalog_manager.cc:5526] T 82151d61cef04050b69f24bc614507fc P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 2 to 5, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:40.745059 21521 leader_election.cc:304] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:40.674777 24593 raft_consensus.cc:2743] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:40.663918 24375 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:40.752847 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:40.795603 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 50634.4us
I20250114 08:59:40.801793 21553 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:40.753249 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.803494 21629 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:40.804638 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.805694 21629 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:40.756191 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.753729 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.754547 21547 rpcz_store.cc:276] 0114 08:59:39.581001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.646318 (+ 65317us) service_pool.cc:225] Handling call
0114 08:59:40.665985 (+1019667us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b540007bcd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:40.666113 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":468559616}
I20250114 08:59:40.806612 21621 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
W20250114 08:59:40.754339 21701 rpcz_store.cc:269] 0114 08:59:38.031801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.036216 (+ 4415us) service_pool.cc:225] Handling call
0114 08:59:40.658942 (+2622726us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54001653d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:40.659093 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171448192}
W20250114 08:59:40.774348 21550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1634) took 1741 ms (client timeout 2148 ms). Trace:
W20250114 08:59:40.775962 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 99185.2us
I20250114 08:59:40.743719 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
W20250114 08:59:40.797093 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 146736us
I20250114 08:59:40.796977 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.807783 21594 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:40.808071 21546 raft_consensus.cc:2388] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
W20250114 08:59:40.808594 21550 rpcz_store.cc:269] 0114 08:59:39.032480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.032601 (+ 121us) service_pool.cc:225] Handling call
0114 08:59:40.774080 (+1741479us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b540031c254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:40.774270 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1057514240}
I20250114 08:59:40.809239 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.810117 21546 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:40.742484 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:40.799085 21667 leader_election.cc:400] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045) with higher term. Message: Invalid argument: T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
I20250114 08:59:40.811357 21667 leader_election.cc:403] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:40.752588 24322 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250114 08:59:40.816350 21668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.037s sys 0.054s
I20250114 08:59:40.808074 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:40.816939 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.817427 21669 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:40.812883 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:40.753352 22268 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
W20250114 08:59:40.818753 21668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.570s user 0.037s sys 0.054s
I20250114 08:59:40.815407 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.799470 24334 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:40.824106 21594 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:40.819226 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 570690us
I20250114 08:59:40.821048 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.832125 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.748406 24025 raft_consensus.cc:2743] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:40.829190 21521 leader_election.cc:400] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:40.836877 21521 leader_election.cc:403] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250114 08:59:40.835651 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.836248 21553 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:40.841269 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.842182 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.830981 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.832820 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.819898 21548 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:40.843725 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.844969 21621 raft_consensus.cc:2363] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:40.846299 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:40.850872 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:40.802592 24619 raft_consensus.cc:3054] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:40.851507 21697 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:40.851552 21629 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:40.853330 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:40.855307 22239 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:40.855954 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.856950 21550 raft_consensus.cc:2388] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 8: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:40.814647 24395 raft_consensus.cc:2798] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:40.856770 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.862710 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.863317 21553 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:40.800658 24094 leader_election.cc:290] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.816195 24322 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:40.864634 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.858290 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:40.754245 22273 rpcz_store.cc:269] 0114 08:59:36.393046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:36.407253 (+ 14207us) service_pool.cc:225] Handling call
0114 08:59:36.510818 (+103565us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa64452c1a0 00007fa64452ba84 00007fa63dbf9c14 00007fa63dbf9ba2
0114 08:59:40.656569 (+4145751us) spinlock_profiling.cc:243] Waited 3.94 s on lock 0x7b54004862d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:40.656783 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9139833344}
I20250114 08:59:40.868207 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:40.863667 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:40.863368 21630 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:40.869324 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.768193 24411 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:40.823232 24334 raft_consensus.cc:491] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:40.871032 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:40.951256 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.957285 21623 raft_consensus.cc:2463] T 82452b55145444529bf0609cd7767adf P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:40.957532 21548 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 10: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:40.954265 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.851500 24033 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 8
I20250114 08:59:40.959301 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:40.963831 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:40.953358 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.859656 24386 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:40.859310 24376 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election won for term 9
W20250114 08:59:40.870460 24468 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.022s sys 0.079s
I20250114 08:59:40.956525 24588 raft_consensus.cc:2798] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:40.862520 24279 raft_consensus.cc:2798] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250114 08:59:40.864573 24527 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.026s sys 0.048s
I20250114 08:59:40.873274 24322 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:40.992615 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:40.870404 24565 raft_consensus.cc:2743] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485) with higher term. Message: Invalid argument: T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:40.993579 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:40.948166 24451 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20250114 08:59:40.991714 21521 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:40.996590 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:40.862785 24424 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:40.991699 24386 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:40.871587 24366 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:40.959142 24547 raft_consensus.cc:2798] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:40.959045 24334 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:40.950223 24395 raft_consensus.cc:695] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:40.895382 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 45503.3us
I20250114 08:59:40.999069 21668 leader_election.cc:304] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:41.050804 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 51807.1us
I20250114 08:59:41.017714 21701 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 3.
I20250114 08:59:41.019129 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:40.963507 24094 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:40.968654 24597 raft_consensus.cc:491] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:41.028617 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.029199 21695 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:40.997503 24468 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s user 0.022s sys 0.080s
I20250114 08:59:41.000842 24588 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:41.032339 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:41.021989 24527 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.028s sys 0.053s
I20250114 08:59:41.080107 21552 raft_consensus.cc:2388] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 6: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:41.038745 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:41.047364 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:40.966795 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:41.022415 24279 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:40.990785 24325 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20250114 08:59:41.073657 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:41.052105 21669 leader_election.cc:336] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.768s (SENT)
W20250114 08:59:41.076447 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 77567.1us
I20250114 08:59:41.079010 21550 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:41.037714 24386 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:41.038343 24376 raft_consensus.cc:695] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:41.088850 22239 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters:
I20250114 08:59:41.033878 24272 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:41.080848 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:41.087893 21669 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:41.081871 21630 raft_consensus.cc:3054] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:41.109580 21668 leader_election.cc:400] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9.
I20250114 08:59:41.115911 21668 leader_election.cc:403] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250114 08:59:41.117728 21699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 2006) took 2513 ms. Trace:
I20250114 08:59:41.118161 21699 rpcz_store.cc:276] 0114 08:59:38.604657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:38.604783 (+ 126us) service_pool.cc:225] Handling call
0114 08:59:38.605128 (+ 345us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:38.638479 (+ 33351us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:38.638526 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:38.639069 (+ 543us) log.cc:844] Serialized 26 byte log entry
0114 08:59:40.466335 (+1827266us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b18003c6460. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:59:40.869109 (+402774us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:40.869147 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.869219 (+ 72us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:41.117417 (+248198us) raft_consensus.cc:1680] finished
0114 08:59:41.117437 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.117662 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20916,"raft.queue_time_us":35043,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":21971,"spinlock_wait_cycles":4126592,"thread_start_us":402608,"threads_started":1,"wal-append.queue_time_us":2460249}
I20250114 08:59:41.119459 24556 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.121862 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:41.122085 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:41.122648 21550 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.123195 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:41.126502 21626 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.015s sys 0.035s
I20250114 08:59:41.038192 24397 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:41.089696 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:41.079989 24529 raft_consensus.cc:2798] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:41.130191 21553 raft_consensus.cc:2388] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 11: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:41.109792 21695 raft_consensus.cc:2463] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:41.131134 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:41.127692 21626 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.015s sys 0.036s
I20250114 08:59:41.128261 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:41.132822 21699 raft_consensus.cc:2388] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 11: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:41.132840 21521 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:41.131850 21623 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:41.115332 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.135736 21621 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:41.084766 24243 raft_consensus.cc:491] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:41.145088 24615 log.cc:926] Time spent T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250114 08:59:41.144786 24243 raft_consensus.cc:513] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.080560 24597 raft_consensus.cc:513] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.145610 24353 leader_election.cc:290] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.146487 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1580) took 1619 ms. Trace:
I20250114 08:59:41.146553 24395 consensus_queue.cc:237] T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.137796 24529 raft_consensus.cc:491] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:41.129709 24272 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.130682 24618 consensus_queue.cc:1035] T fa184c9a07ba40518b2e4d61a40e4da2 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250114 08:59:41.149073 24529 raft_consensus.cc:3054] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:41.108309 24375 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20250114 08:59:41.153012 24547 raft_consensus.cc:491] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:41.153509 24547 raft_consensus.cc:3054] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:41.153697 24594 consensus_meta.cc:287] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.713s user 0.008s sys 0.002s
I20250114 08:59:41.154155 24594 raft_consensus.cc:513] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.086616 24094 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } attrs { replace: true } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.157899 24399 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:41.158069 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 48668.7us
I20250114 08:59:41.158825 21668 leader_election.cc:304] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:41.159660 21668 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:41.159803 24094 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:41.164860 24325 raft_consensus.cc:2798] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:41.165418 24325 raft_consensus.cc:695] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:41.166333 24325 consensus_queue.cc:237] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.168669 24450 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.169118 24450 raft_consensus.cc:491] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:41.169515 24450 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.169832 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:41.170459 21449 catalog_manager.cc:5526] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 5, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 5 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:41.173116 24612 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:41.173938 21552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:41.174369 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.174750 21552 raft_consensus.cc:604] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:41.175129 21552 raft_consensus.cc:612] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:41.175619 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.179785 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:41.156206 24594 leader_election.cc:290] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:41.176267 21625 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:41.083607 24588 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:41.124866 24595 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.140177 24411 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:41.146988 21554 rpcz_store.cc:276] 0114 08:59:39.527013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.656346 (+129333us) service_pool.cc:225] Handling call
0114 08:59:39.704761 (+ 48415us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:39.869247 (+164486us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:39.869277 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:39.881416 (+ 12139us) log.cc:844] Serialized 26 byte log entry
0114 08:59:40.434343 (+552927us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:40.434379 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.434446 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:41.146265 (+711819us) raft_consensus.cc:1680] finished
0114 08:59:41.146286 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.146426 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3179,"raft.queue_time_us":145574,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":4939,"spinlock_wait_cycles":1221376,"thread_start_us":563131,"threads_started":2,"wal-append.queue_time_us":1063012}
I20250114 08:59:41.207373 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.208250 21621 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
I20250114 08:59:41.139765 24376 consensus_queue.cc:237] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.140741 24530 raft_consensus.cc:2798] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:41.147019 24243 leader_election.cc:290] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:41.160449 21668 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:41.149118 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1589) took 1239 ms. Trace:
I20250114 08:59:41.210414 21555 rpcz_store.cc:276] 0114 08:59:39.910048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.910546 (+ 498us) service_pool.cc:225] Handling call
0114 08:59:41.148915 (+1238369us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b540031c754. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:41.149058 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724384640}
I20250114 08:59:41.087054 24279 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:41.095391 24495 consensus_meta.cc:287] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.651s user 0.009s sys 0.000s
I20250114 08:59:41.185525 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:41.037887 24424 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.195207 21518 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:41.195353 21697 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:41.195415 21547 raft_consensus.cc:2463] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 10.
I20250114 08:59:41.197813 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.194243 21550 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:41.238435 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1599) took 1246 ms. Trace:
I20250114 08:59:41.239117 21630 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:41.238927 21550 rpcz_store.cc:276] 0114 08:59:39.992379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.237507 (+1245128us) service_pool.cc:225] Handling call
0114 08:59:41.237831 (+ 324us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.238023 (+ 192us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.238061 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:41.238117 (+ 56us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.238170 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.238239 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.238361 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 08:59:41.198370 21622 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.010s sys 0.016s
I20250114 08:59:41.199851 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:41.240186 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1597) took 1248 ms. Trace:
I20250114 08:59:41.240320 22240 leader_election.cc:304] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:41.240198 21622 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s user 0.010s sys 0.016s
I20250114 08:59:41.240821 21554 rpcz_store.cc:276] 0114 08:59:39.991750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.207091 (+1215341us) service_pool.cc:225] Handling call
0114 08:59:41.239651 (+ 32560us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.239806 (+ 155us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.239842 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:41.239892 (+ 50us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.239936 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.240005 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.240122 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:41.240410 21701 raft_consensus.cc:1808] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:41.243204 21701 raft_consensus.cc:2405] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
W20250114 08:59:41.210354 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 49975.9us
I20250114 08:59:41.221462 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.244196 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.244971 21629 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:41.206408 24595 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:41.251468 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.245988 21668 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:41.254068 21554 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 10. Current term is 11. Ops: []
I20250114 08:59:41.247367 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.252127 21625 raft_consensus.cc:2463] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:41.255344 21668 leader_election.cc:304] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:41.234953 21553 raft_consensus.cc:2463] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 9.
W20250114 08:59:41.247905 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 423873us
I20250114 08:59:41.248502 21630 raft_consensus.cc:2388] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:41.248803 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.219673 24451 raft_consensus.cc:2798] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:41.264508 21621 raft_consensus.cc:2388] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 9: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:41.265424 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.270128 21668 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:41.263564 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:41.288293 21621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1727) took 1022 ms. Trace:
I20250114 08:59:41.281654 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50144 (request call id 211) took 1834 ms. Trace:
I20250114 08:59:41.302850 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:41.304204 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 80762us
I20250114 08:59:41.306244 21621 rpcz_store.cc:276] 0114 08:59:40.265728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.267987 (+1002259us) service_pool.cc:225] Handling call
0114 08:59:41.287673 (+ 19686us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.287915 (+ 242us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.287940 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:41.288003 (+ 63us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:41.288050 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.288120 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.288232 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:41.307498 21625 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 10: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:41.290699 21629 raft_consensus.cc:1270] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:41.308555 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:41.309756 21551 rpcz_store.cc:276] 0114 08:59:39.447106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.447255 (+ 149us) service_pool.cc:225] Handling call
0114 08:59:41.265248 (+1817993us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b540031ae54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:41.281567 (+ 16319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3919757056}
I20250114 08:59:41.310479 21550 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:41.311324 21550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 215) took 1053 ms. Trace:
I20250114 08:59:41.311513 22239 leader_election.cc:304] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:41.311744 21550 rpcz_store.cc:276] 0114 08:59:40.258382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.256379 (+997997us) service_pool.cc:225] Handling call
0114 08:59:41.310266 (+ 53887us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.311050 (+ 784us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.311262 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:41.245725 24549 raft_consensus.cc:2798] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.259207 24411 raft_consensus.cc:2798] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:41.246265 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:41.315650 21668 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:41.316679 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.337855 21630 raft_consensus.cc:2405] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:41.317220 21629 raft_consensus.cc:1212] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Deduplicated request from leader. Original: 4.3->[6.4-8.5] Dedup: 8.5->[]
I20250114 08:59:41.338850 21629 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20250114 08:59:41.319387 21547 raft_consensus.cc:1270] T 3e70c3cb3e8e4656945ec18e88e46101 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250114 08:59:41.339767 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 1651) took 1048 ms. Trace:
I20250114 08:59:41.340173 21547 rpcz_store.cc:276] 0114 08:59:40.291807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.270633 (+978826us) service_pool.cc:225] Handling call
0114 08:59:41.318981 (+ 48348us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.339449 (+ 20468us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.339689 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:41.250594 24530 raft_consensus.cc:695] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:41.341423 21623 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:41.342175 21625 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:41.342725 21623 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20250114 08:59:41.342325 24530 consensus_queue.cc:237] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.267653 24451 raft_consensus.cc:491] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:41.259707 24617 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:41.333838 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:41.260602 24595 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.334440 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.353225 21621 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:41.326845 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:41.303895 24590 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.313539 24559 raft_consensus.cc:2743] T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:41.279665 24375 raft_consensus.cc:2743] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:41.315850 24538 raft_consensus.cc:2798] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:41.364997 24538 raft_consensus.cc:491] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:41.365451 24538 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:41.346639 24451 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:41.287757 24527 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:41.298050 24424 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.355170 21594 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:41.390741 21594 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:41.392443 21594 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:41.358578 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 163430us
I20250114 08:59:41.239238 24495 raft_consensus.cc:513] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.401443 24543 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:41.401911 24543 raft_consensus.cc:491] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:41.402244 24543 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:41.402540 24495 leader_election.cc:290] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.346917 24450 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.406390 24450 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:41.408524 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:41.368330 24322 raft_consensus.cc:513] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.409881 21548 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:41.411835 24630 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.412331 24630 raft_consensus.cc:491] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:41.314535 21553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 217) took 1036 ms. Trace:
I20250114 08:59:41.361198 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.413906 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.414395 24322 leader_election.cc:290] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.359719 24550 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.351776 21624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1534) took 1718 ms. Trace:
I20250114 08:59:41.414901 24550 raft_consensus.cc:491] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:41.415323 24550 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.186300 21449 catalog_manager.cc:5526] T 2d803a1f3141467fa3c2c63696526f84 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 6, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 6 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:41.415047 21624 rpcz_store.cc:276] 0114 08:59:39.633312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.385601 (+752289us) service_pool.cc:225] Handling call
0114 08:59:40.385896 (+ 295us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:40.386091 (+ 195us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:40.870673 (+484582us) log.cc:844] Serialized 14 byte log entry
0114 08:59:41.316671 (+445998us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:41.316720 (+ 49us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:41.316772 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.316845 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.351694 (+ 34849us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":735765,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":357,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":19482,"thread_start_us":753965,"threads_started":2}
I20250114 08:59:41.416723 22272 raft_consensus.cc:1235] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:41.417290 21553 rpcz_store.cc:276] 0114 08:59:40.278572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.312999 (+1034427us) service_pool.cc:225] Handling call
0114 08:59:41.314067 (+ 1068us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.314225 (+ 158us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.314250 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:41.314288 (+ 38us) raft_consensus.cc:1640] Marking committed up to 7
0114 08:59:41.314317 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.314371 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.314473 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:41.417404 21449 catalog_manager.cc:5526] T be5a427c5ff04f9fa27e3e64bd27b146 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 4, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:41.370086 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1722) took 1590 ms. Trace:
I20250114 08:59:41.336208 24564 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.367154 24399 raft_consensus.cc:2743] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539) with higher term. Message: Invalid argument: T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9.
I20250114 08:59:41.394218 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.349872 24411 raft_consensus.cc:695] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:41.397683 24468 raft_consensus.cc:2798] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:41.397799 24094 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250114 08:59:41.398449 24620 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20250114 08:59:41.399284 24622 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250114 08:59:41.460333 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.399706 24597 leader_election.cc:290] T ec8afddecbe8465193b684ed870589f0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:41.430846 24468 raft_consensus.cc:491] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:41.414227 21594 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:41.465054 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 50903.8us
I20250114 08:59:41.466478 24549 raft_consensus.cc:491] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:41.350912 22268 raft_consensus.cc:1808] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Handling vote request from an unknown peer 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:41.357475 24590 raft_consensus.cc:491] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:41.350229 24593 consensus_queue.cc:1046] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20250114 08:59:41.425038 21626 rpcz_store.cc:276] 0114 08:59:39.781302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.495159 (+713857us) service_pool.cc:225] Handling call
0114 08:59:40.495513 (+ 354us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:40.501378 (+ 5865us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:40.501415 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:40.501866 (+ 451us) log.cc:844] Serialized 26 byte log entry
0114 08:59:41.133633 (+631767us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.199856 (+ 66223us) log.cc:844] Serialized 14 byte log entry
0114 08:59:41.229847 (+ 29991us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.229953 (+ 106us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:41.369867 (+139914us) raft_consensus.cc:1680] finished
0114 08:59:41.369887 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.370028 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65937,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":29587,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":12520,"thread_start_us":631288,"threads_started":1,"wal-append.queue_time_us":851494}
I20250114 08:59:41.437822 21550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:41.490315 21550 raft_consensus.cc:604] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:41.490824 21550 raft_consensus.cc:612] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:41.444911 24411 consensus_queue.cc:237] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.419828 21449 catalog_manager.cc:5526] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 4, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250114 08:59:41.395766 24527 raft_consensus.cc:695] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:41.496443 24527 consensus_queue.cc:237] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.422433 24030 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:41.420802 24587 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.153s
I20250114 08:59:41.499812 24587 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:41.500161 24587 raft_consensus.cc:738] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:41.430895 24188 consensus_queue.cc:1046] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20250114 08:59:41.452867 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.457080 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.501022 24587 consensus_queue.cc:260] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.501668 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.501929 21630 raft_consensus.cc:3054] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 10
I20250114 08:59:41.502405 24587 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:41.502951 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.503731 21626 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 8: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
W20250114 08:59:41.458251 24624 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20250114 08:59:41.412761 24630 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.464131 24468 raft_consensus.cc:3054] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.468739 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.476220 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.510021 21624 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:41.477087 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:41.472635 24549 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.486649 22268 raft_consensus.cc:2405] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250114 08:59:41.408229 24588 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.419531 24279 raft_consensus.cc:513] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.498735 21622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1535) took 1865 ms. Trace:
I20250114 08:59:41.489091 24590 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:41.501679 21621 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:41.506047 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.525780 21626 raft_consensus.cc:2388] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 8: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:41.510847 21697 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:41.515919 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.520902 21622 rpcz_store.cc:276] 0114 08:59:39.633735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.396151 (+762416us) service_pool.cc:225] Handling call
0114 08:59:40.438382 (+ 42231us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:40.448788 (+ 10406us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:40.448833 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:40.458480 (+ 9647us) log.cc:844] Serialized 26 byte log entry
0114 08:59:40.653694 (+195214us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b18003dd9e0. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:59:41.241906 (+588212us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.241960 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.242061 (+ 101us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:41.490030 (+247969us) raft_consensus.cc:1680] finished
0114 08:59:41.490057 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.498659 (+ 8602us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18624,"raft.queue_time_us":1457,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":8301,"spinlock_wait_cycles":135121280,"thread_start_us":587964,"threads_started":1,"wal-append.queue_time_us":916112}
I20250114 08:59:41.518476 21627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1532) took 1975 ms. Trace:
I20250114 08:59:41.528630 21624 raft_consensus.cc:2363] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:41.527626 21627 rpcz_store.cc:276] 0114 08:59:39.543354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.372885 (+829531us) service_pool.cc:225] Handling call
0114 08:59:40.398230 (+ 25345us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:40.429989 (+ 31759us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:40.430032 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:40.438288 (+ 8256us) log.cc:844] Serialized 26 byte log entry
0114 08:59:40.954916 (+516628us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:40.954963 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:40.955117 (+ 154us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:41.518227 (+563110us) raft_consensus.cc:1680] finished
0114 08:59:41.518245 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:41.518398 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":275,"raft.queue_time_us":5088,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":2343,"thread_start_us":489946,"threads_started":1,"wal-append.queue_time_us":741622}
I20250114 08:59:41.530632 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.525234 21630 raft_consensus.cc:2463] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
I20250114 08:59:41.436687 24529 raft_consensus.cc:513] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.538676 21624 raft_consensus.cc:2388] T 6c3a3fdbbbca423695a05eb175031e72 P d7732324fb5f48d3b16a29d4bb2584ed [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 11: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:41.543892 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:41.544181 21518 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:41.544646 21621 raft_consensus.cc:2463] T 92fb0c7bd28441aabcf6344f04b84ee6 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 7.
W20250114 08:59:41.545097 21628 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.003s sys 0.011s
I20250114 08:59:41.467569 24386 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.548543 22271 raft_consensus.cc:1235] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20250114 08:59:41.536393 24033 consensus_meta.cc:287] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.546s user 0.009s sys 0.002s
I20250114 08:59:41.551215 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:41.576658 21628 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.539s user 0.003s sys 0.011s
I20250114 08:59:41.576857 21622 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:41.539374 22273 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:41.588928 21700 raft_consensus.cc:1235] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:41.589103 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.588189 21550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:41.590140 21550 raft_consensus.cc:604] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:41.590507 21550 raft_consensus.cc:612] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:41.535614 24617 raft_consensus.cc:2798] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20250114 08:59:41.582489 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 530372us
I20250114 08:59:41.528337 24272 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:41.611030 22241 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:41.614194 22241 leader_election.cc:304] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:41.547736 24322 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:41.623579 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:41.578174 24370 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.999s user 0.014s sys 0.044s
I20250114 08:59:41.579046 24033 raft_consensus.cc:2743] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045) with higher term. Message: Invalid argument: T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8.
I20250114 08:59:41.636504 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.589533 24629 raft_consensus.cc:2743] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:41.695956 21621 raft_consensus.cc:2388] T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 10: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:41.556030 24334 raft_consensus.cc:513] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.709230 24390 raft_consensus.cc:2798] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 08:59:41.709964 24390 raft_consensus.cc:491] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7732324fb5f48d3b16a29d4bb2584ed)
I20250114 08:59:41.710431 24390 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:41.716828 21449 catalog_manager.cc:5526] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 6, leader changed from 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129) to d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130). New cstate: current_term: 6 leader_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } }
I20250114 08:59:41.626513 24525 raft_consensus.cc:2798] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:41.719756 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:41.635061 24370 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.056s user 0.015s sys 0.044s
I20250114 08:59:41.734738 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:41.756350 21626 raft_consensus.cc:3054] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:41.750735 21550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:41.763684 24529 leader_election.cc:290] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:41.767292 24543 raft_consensus.cc:513] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:41.769129 24580 consensus_meta.cc:287] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 1.011s user 0.011s sys 0.001s
I20250114 08:59:41.769680 24580 raft_consensus.cc:513] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:41.771818 24550 raft_consensus.cc:513] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:41.507295 24496 consensus_queue.cc:1159] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" responder_term: 8 status { last_received { term: 7 index: 9 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250114 08:59:41.773145 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.077849 21624 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:41.941841 21703 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 LEADER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20250114 08:59:42.066412 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 449116us
I20250114 08:59:41.774788 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.081030 21622 raft_consensus.cc:2463] T 80026dc5df8c4456a48f734e1a64b930 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8.
I20250114 08:59:41.774905 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.083566 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:41.758671 21550 raft_consensus.cc:604] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:41.965282 21554 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.016s sys 0.019s
W20250114 08:59:42.090086 21554 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.017s sys 0.023s
I20250114 08:59:41.800182 21696 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
W20250114 08:59:41.718765 24619 consensus_meta.cc:287] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.854s user 0.011s sys 0.000s
W20250114 08:59:41.931000 21548 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.013s sys 0.021s
W20250114 08:59:42.093789 21548 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.681s user 0.016s sys 0.026s
W20250114 08:59:42.009912 21668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.013s sys 0.025s
W20250114 08:59:42.094594 21668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.015s sys 0.026s
I20250114 08:59:41.774008 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.090847 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:42.095451 21621 raft_consensus.cc:2371] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10. Re-sending same reply.
I20250114 08:59:42.091815 22241 leader_election.cc:304] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7
W20250114 08:59:42.096170 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 780568us
W20250114 08:59:41.886747 21551 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.511s user 0.016s sys 0.016s
W20250114 08:59:42.092015 21669 leader_election.cc:336] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.608s (SENT)
I20250114 08:59:42.095516 21702 raft_consensus.cc:3054] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:42.099450 21669 leader_election.cc:336] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.036s (SENT)
I20250114 08:59:42.100102 21669 leader_election.cc:304] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:41.790035 21698 raft_consensus.cc:2463] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 8.
I20250114 08:59:41.800212 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.102260 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:42.103067 21697 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:41.909332 22267 raft_consensus.cc:2363] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9.
W20250114 08:59:41.986059 24451 consensus_meta.cc:287] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.617s user 0.010s sys 0.000s
I20250114 08:59:41.810923 24279 leader_election.cc:290] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:42.119642 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.124431 21622 raft_consensus.cc:2463] T 0564bed845d3430e8831ee6428122144 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:41.817840 24334 leader_election.cc:290] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.945096 24580 leader_election.cc:290] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:41.777292 24588 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:42.062865 24593 raft_consensus.cc:3049] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
W20250114 08:59:42.125322 24547 consensus_meta.cc:287] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.968s user 0.012s sys 0.002s
I20250114 08:59:42.136632 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.129891 24411 raft_consensus.cc:2743] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:41.811955 24386 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:42.128363 21669 leader_election.cc:336] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.197s (SENT)
I20250114 08:59:42.215914 21669 leader_election.cc:304] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:42.132844 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.110026 24619 raft_consensus.cc:2743] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 001d4f9f4b5d4124b95dcddb6adac2d6 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10.
W20250114 08:59:42.135699 21623 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.003s sys 0.014s
W20250114 08:59:42.223879 21623 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.003s sys 0.014s
W20250114 08:59:42.193410 24637 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.393s user 0.000s sys 0.001s
W20250114 08:59:42.153584 24319 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.009s sys 0.013s
I20250114 08:59:42.126989 24451 raft_consensus.cc:513] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:42.222169 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.269290 21622 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 6.
I20250114 08:59:42.136221 24593 raft_consensus.cc:738] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:42.224763 21623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1752) took 1255 ms. Trace:
I20250114 08:59:42.112278 24581 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:42.193679 21594 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.021s sys 0.047s
W20250114 08:59:42.340263 21594 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.022s sys 0.047s
W20250114 08:59:42.340732 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 873918us
W20250114 08:59:42.194185 24030 consensus_meta.cc:287] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.695s user 0.010s sys 0.001s
I20250114 08:59:42.201500 21624 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 9.
I20250114 08:59:42.131858 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.362995 22240 leader_election.cc:304] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:42.363566 21621 raft_consensus.cc:2405] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:42.250388 21697 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:41.810949 24550 leader_election.cc:290] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:42.369887 21624 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20250114 08:59:42.262339 21668 leader_election.cc:336] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539): Timed out: RequestConsensusVote RPC to 127.19.231.129:45539 timed out after 1.644s (SENT)
W20250114 08:59:42.371282 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 109005us
I20250114 08:59:41.822075 24595 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.160624 24543 leader_election.cc:290] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:42.198943 24654 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:42.396129 21554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1600) took 2114 ms. Trace:
I20250114 08:59:42.396598 21554 rpcz_store.cc:276] 0114 08:59:40.281982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.305686 (+1023704us) service_pool.cc:225] Handling call
0114 08:59:41.306017 (+ 331us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:41.348872 (+ 42855us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.348914 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:41.393639 (+ 44725us) log.cc:844] Serialized 26 byte log entry
0114 08:59:42.090635 (+696996us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:42.090693 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.090803 (+ 110us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:42.395849 (+305046us) raft_consensus.cc:1680] finished
0114 08:59:42.395878 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.396054 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64383,"raft.queue_time_us":23390,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":9700,"thread_start_us":643667,"threads_started":1,"wal-append.queue_time_us":826404}
I20250114 08:59:42.395401 24654 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:42.406996 24637 log.cc:926] Time spent T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20250114 08:59:42.408164 24640 log.cc:926] Time spent T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
I20250114 08:59:41.799804 24525 raft_consensus.cc:695] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:42.299589 21623 rpcz_store.cc:276] 0114 08:59:40.969501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.343444 (+373943us) service_pool.cc:225] Handling call
0114 08:59:41.343725 (+ 281us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.343928 (+ 203us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 08:59:41.345943 (+ 2015us) log.cc:844] Serialized 14 byte log entry
0114 08:59:42.224299 (+878356us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:42.224369 (+ 70us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:42.224437 (+ 68us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.224541 (+ 104us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.224697 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1391,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":586686,"thread_start_us":877942,"threads_started":2,"wal-append.queue_time_us":595880}
I20250114 08:59:42.370164 21549 raft_consensus.cc:2405] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250114 08:59:42.149258 24638 raft_consensus.cc:2743] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 08:59:42.392999 24279 raft_consensus.cc:2798] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election won for term 9
W20250114 08:59:42.393546 24639 log.cc:926] Time spent T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
W20250114 08:59:42.394146 24634 log.cc:926] Time spent T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.387s user 0.000s sys 0.001s
I20250114 08:59:42.239596 24597 raft_consensus.cc:2743] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:42.177492 24547 raft_consensus.cc:513] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:41.734773 24629 consensus_queue.cc:1046] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.615s
I20250114 08:59:42.412523 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1606) took 1573 ms. Trace:
I20250114 08:59:42.595891 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.079392 21449 catalog_manager.cc:5526] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 4, leader changed from cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
W20250114 08:59:42.598212 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 469834us
W20250114 08:59:42.400285 24630 consensus_meta.cc:287] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.877s user 0.009s sys 0.000s
I20250114 08:59:42.596619 21547 rpcz_store.cc:276] 0114 08:59:40.839506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.340885 (+501379us) service_pool.cc:225] Handling call
0114 08:59:41.341259 (+ 374us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:41.349187 (+ 7928us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.349226 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:41.355826 (+ 6600us) log.cc:844] Serialized 26 byte log entry
0114 08:59:41.924777 (+568951us) raft_consensus.cc:1640] Marking committed up to 3
0114 08:59:41.924816 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.924895 (+ 79us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:42.150496 (+225601us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b4800080058. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa644491581 00007fa644488677 00007fa64448850a 00007fa64457c9bf 00007fa64457c687 00007fa644590b8c 00007fa644590b38 00007fa644590ab8 00007fa644590a73 00007fa64458fd1c 00007fa6434fcd16
0114 08:59:42.412085 (+261589us) raft_consensus.cc:1680] finished
0114 08:59:42.412134 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.412431 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25629,"raft.queue_time_us":8120,"raft.run_cpu_time_us":12923,"raft.run_wall_time_us":816871,"spinlock_wait_cycles":9473920,"thread_start_us":499781,"threads_started":1,"wal-append.queue_time_us":918158}
I20250114 08:59:42.586441 21551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 218) took 2183 ms. Trace:
W20250114 08:59:42.413224 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 71686.1us
I20250114 08:59:42.632499 21551 rpcz_store.cc:276] 0114 08:59:40.429092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.329617 (+900525us) service_pool.cc:225] Handling call
0114 08:59:41.351516 (+ 21899us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.351753 (+ 237us) raft_consensus.cc:1529] Early marking committed up to index 10
0114 08:59:42.097361 (+745608us) log.cc:844] Serialized 14 byte log entry
0114 08:59:42.585902 (+488541us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:42.585991 (+ 89us) raft_consensus.cc:1640] Marking committed up to 10
0114 08:59:42.586069 (+ 78us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.586177 (+ 108us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.586356 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":741379,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"thread_start_us":1169058,"threads_started":2}
W20250114 08:59:42.568228 21626 consensus_meta.cc:287] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.811s user 0.011s sys 0.000s
I20250114 08:59:42.634600 21626 raft_consensus.cc:2463] T a93e2235a71d41a4a46a1e50fd8c02b7 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:42.635922 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.637388 21626 raft_consensus.cc:2463] T 2d803a1f3141467fa3c2c63696526f84 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:42.640528 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:42.643494 21669 leader_election.cc:336] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.522s (SENT)
W20250114 08:59:42.644956 21669 leader_election.cc:336] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.654s (SENT)
I20250114 08:59:42.646392 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
W20250114 08:59:42.410395 24642 log.cc:926] Time spent T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.370s user 0.000s sys 0.001s
I20250114 08:59:42.409183 24654 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:42.590430 24411 consensus_queue.cc:1035] T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.100s
I20250114 08:59:42.533437 24334 consensus_queue.cc:1035] T 3e70c3cb3e8e4656945ec18e88e46101 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.156s
I20250114 08:59:42.800371 22273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48992 (request call id 267) took 1714 ms. Trace:
I20250114 08:59:42.649708 24630 raft_consensus.cc:513] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:42.810740 21625 raft_consensus.cc:2388] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
W20250114 08:59:42.681169 22241 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.013s sys 0.025s
W20250114 08:59:42.811492 22241 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.703s user 0.013s sys 0.056s
W20250114 08:59:42.802063 21549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33886 (request call id 1593) took 2811 ms (client timeout 1644 ms). Trace:
I20250114 08:59:42.647837 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.671748 21621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1"
dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:55318
I20250114 08:59:42.614711 24604 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 08:59:42.807746 21521 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.008s sys 0.016s
I20250114 08:59:42.758790 21552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1596) took 2767 ms. Trace:
I20250114 08:59:42.817539 21700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 2051) took 1637 ms. Trace:
W20250114 08:59:42.815266 21698 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45060 (request call id 216) took 1969 ms (client timeout 2163 ms). Trace:
I20250114 08:59:42.727013 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:42.636394 24590 consensus_meta.cc:287] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.097s user 0.013s sys 0.000s
W20250114 08:59:42.749063 21702 consensus_meta.cc:287] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.647s user 0.012s sys 0.000s
I20250114 08:59:42.865967 21626 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:42.866621 21702 raft_consensus.cc:2463] T 2e77833a006f478e86764142c505ab6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 9.
I20250114 08:59:42.813244 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.864645 21552 rpcz_store.cc:276] 0114 08:59:39.991408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.175888 (+1184480us) service_pool.cc:225] Handling call
0114 08:59:41.176227 (+ 339us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:41.198475 (+ 22248us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.198519 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:41.212358 (+ 13839us) log.cc:844] Serialized 26 byte log entry
0114 08:59:41.591694 (+379336us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.591747 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.591836 (+ 89us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:42.758567 (+1166731us) raft_consensus.cc:1680] finished
0114 08:59:42.758590 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.758728 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8329,"raft.queue_time_us":9648,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6473,"thread_start_us":331692,"threads_started":1,"wal-append.queue_time_us":838628}
I20250114 08:59:42.867624 21629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1777) took 1353 ms. Trace:
W20250114 08:59:42.863061 21521 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.721s user 0.008s sys 0.019s
I20250114 08:59:42.865964 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:42.688050 21548 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.004s sys 0.035s
W20250114 08:59:42.868950 21548 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.005s sys 0.044s
W20250114 08:59:42.836647 21549 rpcz_store.cc:269] 0114 08:59:39.990419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.365263 (+374844us) service_pool.cc:225] Handling call
0114 08:59:42.801774 (+2436511us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b540024f9d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.801941 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4463121152}
W20250114 08:59:42.757040 24652 log.cc:926] Time spent T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20250114 08:59:42.732110 21704 raft_consensus.cc:2463] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 10.
W20250114 08:59:42.759305 24627 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 1.466s user 0.000s sys 0.020s
W20250114 08:59:42.646909 21669 leader_election.cc:336] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.657s (SENT)
I20250114 08:59:42.799139 24564 raft_consensus.cc:491] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:42.854930 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:42.839370 21624 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:42.715023 24641 log.cc:926] Time spent T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.599s user 0.000s sys 0.001s
W20250114 08:59:42.836390 24538 consensus_meta.cc:287] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 1.470s user 0.013s sys 0.000s
I20250114 08:59:42.890679 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.891352 21626 raft_consensus.cc:3054] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:42.892027 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:42.892639 21552 raft_consensus.cc:3054] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:42.814620 24630 leader_election.cc:290] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:42.887523 21624 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
W20250114 08:59:42.865679 21698 rpcz_store.cc:269] 0114 08:59:40.845812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.845977 (+ 165us) service_pool.cc:225] Handling call
0114 08:59:42.815056 (+1969079us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b54002b0a54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.815178 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2009778432}
I20250114 08:59:42.409646 24656 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:42.573076 24593 consensus_queue.cc:260] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.895538 22240 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:42.895709 24656 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:42.896404 24668 raft_consensus.cc:2798] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:42.897039 24668 raft_consensus.cc:695] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [term 7 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:42.896888 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.897929 24656 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:42.897938 21624 raft_consensus.cc:2371] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6. Re-sending same reply.
I20250114 08:59:42.898111 24668 consensus_queue.cc:237] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.868005 21629 rpcz_store.cc:276] 0114 08:59:41.514150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:42.812908 (+1298758us) service_pool.cc:225] Handling call
0114 08:59:42.867564 (+ 54656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 08:59:42.899317 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.900049 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.900496 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.901079 21629 raft_consensus.cc:2463] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:42.840646 22241 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:42.645844 24279 raft_consensus.cc:695] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [term 9 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:42.603440 24595 leader_election.cc:290] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:42.902213 24595 raft_consensus.cc:2798] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election won for term 6
W20250114 08:59:42.902274 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 61733.8us
I20250114 08:59:42.902802 24595 raft_consensus.cc:695] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:42.902436 24279 consensus_queue.cc:237] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.903741 24595 consensus_queue.cc:237] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.871014 21551 raft_consensus.cc:1270] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250114 08:59:42.838218 21621 raft_consensus.cc:604] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:42.896092 24593 raft_consensus.cc:3054] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:42.872079 21669 leader_election.cc:304] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:42.873260 21547 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:42.908834 21628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57320 (request call id 1526) took 3590 ms. Trace:
I20250114 08:59:42.909739 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:42.910354 21547 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:42.813881 24188 raft_consensus.cc:3049] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:42.912032 24188 raft_consensus.cc:738] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:42.914561 21449 catalog_manager.cc:5526] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 4 to 5, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:42.915666 24188 consensus_queue.cc:260] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.809991 22273 rpcz_store.cc:276] 0114 08:59:41.086048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.087465 (+ 1417us) service_pool.cc:225] Handling call
0114 08:59:41.124568 (+ 37103us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.611474 (+486906us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.799998 (+1188524us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b5400670554. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:42.800260 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":905694464}
I20250114 08:59:42.876186 21703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46398 (request call id 2039) took 2021 ms. Trace:
I20250114 08:59:42.876888 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:42.877365 24549 consensus_meta.cc:287] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.349s user 0.007s sys 0.005s
I20250114 08:59:42.877794 24547 leader_election.cc:290] T d0b4ce684a6943809542800e14d94079 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:42.872815 24390 consensus_meta.cc:287] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 1.162s user 0.013s sys 0.000s
I20250114 08:59:42.878639 24564 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:42.878858 22271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49004 (request call id 454) took 1560 ms. Trace:
I20250114 08:59:42.879043 21555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1598) took 2887 ms. Trace:
I20250114 08:59:42.876261 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b4ce684a6943809542800e14d94079" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:42.881035 21546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 1658) took 2505 ms (client timeout 1835 ms). Trace:
I20250114 08:59:42.650316 24315 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:42.893842 24538 raft_consensus.cc:513] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.872097 24604 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:42.901144 21624 raft_consensus.cc:2371] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 8. Re-sending same reply.
I20250114 08:59:42.905624 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:42.906145 21621 raft_consensus.cc:612] T addec468f1bd4f19a98a8d5ce1222fc1 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250114 08:59:42.909267 21628 rpcz_store.cc:276] 0114 08:59:39.318327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:39.468229 (+149902us) service_pool.cc:225] Handling call
0114 08:59:39.468532 (+ 303us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 08:59:39.508548 (+ 40016us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:39.508578 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 08:59:39.530744 (+ 22166us) log.cc:844] Serialized 52 byte log entry
0114 08:59:40.555858 (+1025114us) spinlock_profiling.cc:243] Waited 931 us on lock 0x7b18000a1200. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa64530e931 00007fa64531b831 00007fa63c676565 00007fa6452887f9 00007fa63c693a6b 00007fa63c691e6f 00007fa63c693fd2 00007fa63f249e4b 00007fa63f24f913 00007fa63f24fcc1 00007fa63f5a145f
0114 08:59:41.017832 (+461974us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.590901 (+573069us) log.cc:844] Serialized 14 byte log entry
0114 08:59:42.111455 (+520554us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.111562 (+ 107us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:42.758759 (+647197us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b440002494c. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c5a57c5 00007fa63c5a61e2 00007fa63f25bda1 00007fa63f25768a 00007fa63f24b4ae 00007fa63f24a7f6 00007fa63f24a418 00007fa63f253df1 00007fa63f253da9 00007fa63f253d39
0114 08:59:42.908557 (+149798us) raft_consensus.cc:1680] finished
0114 08:59:42.908587 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.908742 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107669,"prepare.queue_time_us":104351,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":642343,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":59797,"spinlock_wait_cycles":39088896,"thread_start_us":1481479,"threads_started":3,"wal-append.queue_time_us":1750341}
W20250114 08:59:42.868213 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 725996us
I20250114 08:59:42.953486 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.871059 21548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33886 (request call id 1609) took 2019 ms. Trace:
I20250114 08:59:42.926087 21622 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
W20250114 08:59:42.957134 24587 consensus_meta.cc:287] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 1.454s user 0.010s sys 0.004s
I20250114 08:59:42.957248 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.958221 24668 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:42.958674 21622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.696087 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.958968 24668 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.961613 21628 raft_consensus.cc:2388] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:42.959830 21623 raft_consensus.cc:2363] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9.
I20250114 08:59:42.864603 21700 rpcz_store.cc:276] 0114 08:59:41.179712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.180630 (+ 918us) service_pool.cc:225] Handling call
0114 08:59:41.180906 (+ 276us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.589365 (+408459us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.817232 (+1227867us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400189cd4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:42.817455 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":937905408}
I20250114 08:59:42.926841 21703 rpcz_store.cc:276] 0114 08:59:40.855171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.964739 (+109568us) service_pool.cc:225] Handling call
0114 08:59:40.997807 (+ 33068us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:42.081889 (+1084082us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.875942 (+794053us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b5400371ed4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:42.876130 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2170180864}
W20250114 08:59:42.776757 24468 consensus_meta.cc:287] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 1.241s user 0.003s sys 0.009s
W20250114 08:59:42.928426 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 281592us
I20250114 08:59:42.931978 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9690a0a2cde84c5d9312c8694185a0b2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:42.964135 21669 leader_election.cc:336] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.688s (SENT)
I20250114 08:59:42.964280 21549 raft_consensus.cc:3054] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:42.917022 24188 raft_consensus.cc:3054] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:42.937008 22274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49004 (request call id 455) took 1603 ms. Trace:
I20250114 08:59:43.002234 22274 rpcz_store.cc:276] 0114 08:59:41.333885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.334231 (+ 346us) service_pool.cc:225] Handling call
0114 08:59:41.519732 (+185501us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.519912 (+ 180us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.519945 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:41.519980 (+ 35us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:41.520012 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.520064 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.936789 (+1416725us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b540062fa50. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cca09 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.936924 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":374152448}
I20250114 08:59:42.950141 21449 catalog_manager.cc:5526] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20250114 08:59:42.939960 22271 rpcz_store.cc:276] 0114 08:59:41.318747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.318924 (+ 177us) service_pool.cc:225] Handling call
0114 08:59:41.352754 (+ 33830us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:41.552900 (+200146us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.878618 (+1325718us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540059b0d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:42.878802 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449404160}
W20250114 08:59:42.940317 21546 rpcz_store.cc:269] 0114 08:59:40.376116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.832521 (+456405us) service_pool.cc:225] Handling call
0114 08:59:42.880857 (+2048336us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540028e1d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.880967 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816508672}
I20250114 08:59:43.006187 21546 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:43.004776 21449 catalog_manager.cc:5526] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 5 to 6, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 6 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
I20250114 08:59:43.007396 22270 raft_consensus.cc:1235] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 9. Current term is 10. Ops: []
I20250114 08:59:43.013792 24653 consensus_queue.cc:1046] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 08:59:43.014748 24653 raft_consensus.cc:3049] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:43.015075 24653 raft_consensus.cc:738] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:42.940018 21555 rpcz_store.cc:276] 0114 08:59:39.992058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.211611 (+1219553us) service_pool.cc:225] Handling call
0114 08:59:41.246725 (+ 35114us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:41.347948 (+101223us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:41.347974 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:41.355517 (+ 7543us) log.cc:844] Serialized 26 byte log entry
0114 08:59:41.836705 (+481188us) raft_consensus.cc:1640] Marking committed up to 4
0114 08:59:41.836762 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:41.836871 (+ 109us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:42.878858 (+1041987us) raft_consensus.cc:1680] finished
0114 08:59:42.878877 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.878991 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":141396,"raft.queue_time_us":101260,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6365,"thread_start_us":427875,"threads_started":1,"wal-append.queue_time_us":1354332}
I20250114 08:59:42.916952 24617 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:43.016144 24617 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:42.874342 24654 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:42.947641 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.372602 24581 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:42.950634 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:42.715580 24451 leader_election.cc:290] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:42.964570 21669 leader_election.cc:304] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:42.785290 24525 consensus_queue.cc:237] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.966042 21626 raft_consensus.cc:2463] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:42.969722 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:42.971438 24336 raft_consensus.cc:491] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:42.937088 24376 raft_consensus.cc:2743] T 2d803a1f3141467fa3c2c63696526f84 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:43.048622 21554 raft_consensus.cc:2405] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250114 08:59:43.049466 21623 raft_consensus.cc:2363] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9.
W20250114 08:59:42.972872 22267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48992 (request call id 266) took 2238 ms (client timeout 1915 ms). Trace:
I20250114 08:59:42.973591 24657 raft_consensus.cc:3049] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 LEADER]: Stepping down as leader of term 10
I20250114 08:59:42.975205 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.975983 24538 leader_election.cc:290] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:42.968138 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.053900 21628 raft_consensus.cc:2463] T 6a451ff14593400ca08caa1b5ed169f2 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 5.
I20250114 08:59:43.053998 21621 raft_consensus.cc:2463] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:42.955706 21548 rpcz_store.cc:276] 0114 08:59:40.851410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.411636 (+560226us) service_pool.cc:225] Handling call
0114 08:59:41.411997 (+ 361us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:41.412186 (+ 189us) raft_consensus.cc:1529] Early marking committed up to index 5
0114 08:59:42.094479 (+682293us) log.cc:844] Serialized 14 byte log entry
0114 08:59:42.870758 (+776279us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 08:59:42.870806 (+ 48us) raft_consensus.cc:1640] Marking committed up to 5
0114 08:59:42.870852 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:42.870909 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:42.871010 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1269,"prepare.queue_time_us":720304,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":3349,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":15320192,"thread_start_us":1456459,"threads_started":2,"wal-append.queue_time_us":662778}
W20250114 08:59:42.979149 24647 log.cc:926] Time spent T addec468f1bd4f19a98a8d5ce1222fc1 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20250114 08:59:42.939529 24630 consensus_queue.cc:1046] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20250114 08:59:42.950131 24315 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:42.985050 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 1.35586e+06us
I20250114 08:59:42.929101 24549 raft_consensus.cc:513] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.074555 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abedb4afaad04af9bbbb7d4266fd2632" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:42.945649 24390 raft_consensus.cc:513] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:42.936573 24656 consensus_queue.cc:1035] T ec8afddecbe8465193b684ed870589f0 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
W20250114 08:59:42.989490 21625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1770) took 1538 ms (client timeout 1695 ms). Trace:
I20250114 08:59:43.078058 24668 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:43.078214 21625 rpcz_store.cc:269] 0114 08:59:41.451273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.460014 (+ 8741us) service_pool.cc:225] Handling call
0114 08:59:42.989205 (+1529191us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003bb5d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.989383 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3103246720}
I20250114 08:59:43.063804 24243 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:43.079484 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.079532 24243 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.081017 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada199404e61479593fa1b64c574bcb8" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.081895 21548 raft_consensus.cc:3054] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:43.048792 24654 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.082263 24243 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:43.064107 24631 consensus_queue.cc:1046] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20250114 08:59:43.083813 24631 raft_consensus.cc:3049] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Stepping down as leader of term 5
I20250114 08:59:43.084043 24654 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:42.997676 24660 log.cc:926] Time spent T cc5506492f514407a5097b5b2e8c7042 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.301s user 0.003s sys 0.000s
W20250114 08:59:43.014981 21696 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 2050) took 1835 ms (client timeout 1660 ms). Trace:
I20250114 08:59:42.942590 21547 raft_consensus.cc:2463] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:43.021430 21629 raft_consensus.cc:2371] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8. Re-sending same reply.
I20250114 08:59:43.041411 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:43.041687 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 77586.2us
I20250114 08:59:43.054773 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.055830 21549 raft_consensus.cc:2463] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:43.016180 24653 consensus_queue.cc:260] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.027045 24468 raft_consensus.cc:513] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.134766 24654 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:43.112375 24564 raft_consensus.cc:513] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.147082 24654 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:43.147765 21696 rpcz_store.cc:269] 0114 08:59:41.179389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.179524 (+ 135us) service_pool.cc:225] Handling call
0114 08:59:43.014740 (+1835216us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400050854. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:43.014883 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":656574208}
I20250114 08:59:43.065074 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:42.870434 24590 raft_consensus.cc:513] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:43.151055 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.151707 21624 raft_consensus.cc:2463] T 6e6879e06c934d438e3ca9c3f093d4be P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 10.
I20250114 08:59:42.985638 21704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46398 (request call id 2041) took 2129 ms. Trace:
I20250114 08:59:43.071895 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b369b35736c413ca069db061ed36bc5" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.022567 24619 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:43.153084 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.153337 24619 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.154469 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.155061 21624 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:43.155828 24619 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:43.024212 24670 raft_consensus.cc:491] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:43.007648 21449 catalog_manager.cc:5526] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e reported cstate change: term changed from 3 to 4, leader changed from d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130) to 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129). New cstate: current_term: 4 leader_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } health_report { overall_health: HEALTHY } } }
W20250114 08:59:42.988126 24666 log.cc:926] Time spent T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20250114 08:59:43.076143 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.156271 24654 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:43.157081 21621 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:43.158010 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.158363 21627 raft_consensus.cc:2388] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:43.159166 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.159361 24468 leader_election.cc:290] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.084208 24631 raft_consensus.cc:738] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
W20250114 08:59:43.131477 21519 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.544s user 0.012s sys 0.063s
I20250114 08:59:43.159799 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.160559 24631 consensus_queue.cc:260] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.161233 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.161896 24631 raft_consensus.cc:3054] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:43.084678 24376 consensus_queue.cc:1046] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250114 08:59:43.064646 24657 raft_consensus.cc:738] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.086205 24315 leader_election.cc:290] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:42.990587 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.163436 24376 raft_consensus.cc:3049] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:43.163775 21551 raft_consensus.cc:3054] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:43.065160 24336 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.163919 24657 consensus_queue.cc:260] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.166122 24657 raft_consensus.cc:3054] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:43.166293 24336 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:43.088024 24580 raft_consensus.cc:491] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:43.167135 24580 raft_consensus.cc:513] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.062060 21552 raft_consensus.cc:2463] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 6.
I20250114 08:59:43.169174 24580 leader_election.cc:290] T f466e3efd6cb41329852e9028490bdf7 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.170250 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:43.089784 21630 raft_consensus.cc:2388] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 7: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:43.136758 24617 raft_consensus.cc:513] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:43.171880 21630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57318 (request call id 1787) took 1397 ms (client timeout 1664 ms). Trace:
I20250114 08:59:43.172338 21699 raft_consensus.cc:2388] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 11: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
W20250114 08:59:43.172232 21630 rpcz_store.cc:269] 0114 08:59:41.774360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.774533 (+ 173us) service_pool.cc:225] Handling call
0114 08:59:43.171677 (+1397144us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54003b92d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:43.171820 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2270902400}
I20250114 08:59:43.173213 24617 leader_election.cc:290] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
W20250114 08:59:43.160326 21669 leader_election.cc:336] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.703s (SENT)
I20250114 08:59:43.173468 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.090481 22270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49004 (request call id 456) took 1737 ms. Trace:
W20250114 08:59:43.174266 21669 leader_election.cc:336] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.843s (SENT)
I20250114 08:59:43.162400 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.090659 24549 leader_election.cc:290] T a03f2f2f70724b85b8b23ed7bc72fbe1 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.091732 24390 leader_election.cc:290] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 4 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.091940 21622 raft_consensus.cc:2405] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20250114 08:59:43.092327 24630 consensus_queue.cc:1225] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer 28a83b7a7d9d4aa899b425086aa45a8e log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250114 08:59:43.092839 22274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77833a006f478e86764142c505ab6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:43.094694 24581 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.095593 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:43.096684 24530 raft_consensus.cc:988] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:43.096930 22239 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:43.185885 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 89031.3us
I20250114 08:59:43.076905 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4913bb74ad4fab98dd737eb0bd5965" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.098816 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:43.100298 21548 raft_consensus.cc:2463] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 7.
I20250114 08:59:43.052918 24643 raft_consensus.cc:2743] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:43.053522 22267 rpcz_store.cc:269] 0114 08:59:40.734570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.739102 (+ 4532us) service_pool.cc:225] Handling call
0114 08:59:42.972567 (+2233465us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b54007003d4. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.972769 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2262666880}
I20250114 08:59:43.149096 24564 leader_election.cc:290] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.149644 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.149935 21626 raft_consensus.cc:2363] T a14dbff33afe4ab483f01265cb35d676 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7.
I20250114 08:59:43.190783 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:43.191190 21624 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:43.192696 21624 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:43.151758 24590 leader_election.cc:290] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Requested vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539)
I20250114 08:59:43.070689 21555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50"
dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
mode: GRACEFUL
new_leader_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:52970
I20250114 08:59:43.152397 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:43.087368 21625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.202340 21625 raft_consensus.cc:2463] T be5a427c5ff04f9fa27e3e64bd27b146 P d7732324fb5f48d3b16a29d4bb2584ed [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 4.
I20250114 08:59:43.173512 21551 raft_consensus.cc:2463] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:43.174377 22270 rpcz_store.cc:276] 0114 08:59:41.353120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.353267 (+ 147us) service_pool.cc:225] Handling call
0114 08:59:41.375197 (+ 21930us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:43.007986 (+1632789us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:43.090267 (+ 82281us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54003d4354. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:43.090424 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3753081088}
I20250114 08:59:43.090394 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:43.175247 21669 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:43.205883 21669 leader_election.cc:336] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.845s (SENT)
W20250114 08:59:43.207404 21669 leader_election.cc:336] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 2.034s (SENT)
I20250114 08:59:43.148116 21628 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Advancing to term 8
I20250114 08:59:43.152668 21704 rpcz_store.cc:276] 0114 08:59:40.855726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:40.970320 (+114594us) service_pool.cc:225] Handling call
0114 08:59:42.985416 (+2015096us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400129b54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5d12ae 00007fa64452bc17 00007fa63dbf9c14 00007fa63dbf9ba2 00007fa63dbf9ad1 00007fa63dbf9a4c 00007fa63dbf8d02 00007fa63d9f8dfc 00007fa63d9f8236 00007fa63d9f7bdf
0114 08:59:42.985569 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2778350336}
I20250114 08:59:43.179661 21695 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 08:59:43.210346 21521 leader_election.cc:304] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
I20250114 08:59:43.088858 24604 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.182031 21630 raft_consensus.cc:2405] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 08:59:43.181880 21627 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:43.153012 21623 raft_consensus.cc:2463] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:43.159833 21621 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 5.
W20250114 08:59:43.188629 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 1.19979e+06us
I20250114 08:59:43.214033 22240 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.201689 21555 raft_consensus.cc:604] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Received request to transfer leadership to TS cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:43.207834 21669 leader_election.cc:304] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:43.175643 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 185530us
I20250114 08:59:43.223409 21549 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 10. Current term is 11. Ops: []
I20250114 08:59:43.232928 21546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.234176 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:43.235018 21552 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:43.236223 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.237035 21547 raft_consensus.cc:3054] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:43.243618 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.245280 21549 raft_consensus.cc:2388] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed in current term 7: Already voted for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in this term.
I20250114 08:59:43.228823 22268 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:43.237635 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:43.247478 21554 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
W20250114 08:59:43.247880 21520 leader_election.cc:336] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.670s (SENT)
I20250114 08:59:43.247908 22239 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.248544 21520 leader_election.cc:304] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.248338 21627 raft_consensus.cc:1270] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250114 08:59:43.255661 21623 raft_consensus.cc:1235] T eb00de7db5024821adbfdfd3f8a0f475 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:43.216048 24617 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:43.190461 24580 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 08:59:43.201581 24619 raft_consensus.cc:491] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:43.265785 21594 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.163918 24376 raft_consensus.cc:738] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:43.268086 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:43.217461 24530 raft_consensus.cc:1076] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Signalling peer cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
W20250114 08:59:43.261853 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 44896.2us
I20250114 08:59:43.216605 24653 raft_consensus.cc:3054] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:43.261094 21624 raft_consensus.cc:1270] T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:43.230430 24395 raft_consensus.cc:2798] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:43.282325 21593 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.271405 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:43.293254 21555 raft_consensus.cc:3054] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:43.272888 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:43.294169 21552 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:43.277534 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.262382 24596 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 57f9cc1dd3c64647848a33a65b08c893 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250114 08:59:43.229111 24390 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:43.281711 20382 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250114 08:59:43.298792 20382 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 08:59:43.299466 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:43.269758 22271 raft_consensus.cc:1235] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:43.283690 24619 raft_consensus.cc:513] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.284174 24334 raft_consensus.cc:3049] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Stepping down as leader of term 7
I20250114 08:59:43.285059 24376 consensus_queue.cc:260] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.286484 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.291258 21594 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:43.292068 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 237424us
W20250114 08:59:43.295209 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 47380.4us
I20250114 08:59:43.295218 21623 raft_consensus.cc:2463] T 106dea9d52554552a1b3e20e7c82a3a8 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:43.254788 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:43.279204 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.316373 22269 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:43.336040 21621 raft_consensus.cc:2388] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 8: Already voted for candidate d7732324fb5f48d3b16a29d4bb2584ed in this term.
I20250114 08:59:43.278928 21630 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250114 08:59:43.340360 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
W20250114 08:59:43.347335 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 90676.7us
I20250114 08:59:43.347405 21626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:43.334174 21629 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:43.341361 21624 raft_consensus.cc:2463] T ee02341e5e574d55aac7eb9b3df9dc73 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:43.183923 24670 raft_consensus.cc:513] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:43.351966 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 60815.8us
W20250114 08:59:43.351922 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 69673.4us
I20250114 08:59:43.356324 21594 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.293339 24322 raft_consensus.cc:2743] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250114 08:59:43.359783 21520 leader_election.cc:336] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.635s (SENT)
I20250114 08:59:43.362701 21520 leader_election.cc:304] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.362951 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.359601 21623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426decde999b4722a874569acae5664f" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.363821 21630 raft_consensus.cc:2371] T a1aeae7c29264f839189001364c51746 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7. Re-sending same reply.
I20250114 08:59:43.364020 21623 raft_consensus.cc:2463] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:43.300734 24654 raft_consensus.cc:2798] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:43.364790 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.366773 21547 raft_consensus.cc:2463] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:43.371688 21621 raft_consensus.cc:2463] T cc9f4dcde58e4d508d5ffbbf301a7a6b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:43.358784 24581 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:43.373062 24581 raft_consensus.cc:491] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:43.373521 24581 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:43.375625 21629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" is_pre_election: true
I20250114 08:59:43.371148 21594 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.275359 24564 consensus_queue.cc:1046] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.085s
I20250114 08:59:43.310691 24617 raft_consensus.cc:2743] T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:43.333530 24334 raft_consensus.cc:738] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.310307 24390 raft_consensus.cc:491] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
W20250114 08:59:43.397147 21520 leader_election.cc:336] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.645s (SENT)
I20250114 08:59:43.397665 21520 leader_election.cc:304] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:43.399765 21520 leader_election.cc:336] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.655s (SENT)
W20250114 08:59:43.400537 21520 connection.cc:232] RPC call timeout handler was delayed by -1.1196350309999161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20250114 08:59:43.401070 21520 leader_election.cc:336] T a14dbff33afe4ab483f01265cb35d676 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.674s (SENT)
W20250114 08:59:43.402218 21520 connection.cc:232] RPC call timeout handler was delayed by -1.0971350309998797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20250114 08:59:43.402820 21520 connection.cc:232] RPC call timeout handler was delayed by -1.0971350309998797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
I20250114 08:59:43.403920 21555 raft_consensus.cc:2463] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:43.405097 21547 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:43.359107 24468 raft_consensus.cc:2798] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:43.358928 24395 raft_consensus.cc:695] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:43.405635 21594 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:43.409353 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 192703us
W20250114 08:59:43.412086 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 204661us
I20250114 08:59:43.380093 24376 raft_consensus.cc:3054] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:43.380534 24549 raft_consensus.cc:2798] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:43.410660 21594 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
W20250114 08:59:43.415038 21520 leader_election.cc:336] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.694s (SENT)
I20250114 08:59:43.417505 21518 leader_election.cc:304] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:43.421265 21520 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.388688 24630 raft_consensus.cc:3049] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 LEADER]: Stepping down as leader of term 9
I20250114 08:59:43.423570 21704 raft_consensus.cc:1270] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20250114 08:59:43.426348 21520 leader_election.cc:336] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891): Timed out: RequestConsensusVote RPC to 127.19.231.130:36891 timed out after 1.695s (SENT)
I20250114 08:59:43.361646 24336 consensus_queue.cc:1035] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.048s
I20250114 08:59:43.408951 21703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:46398
I20250114 08:59:43.396322 24390 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:43.392701 24654 raft_consensus.cc:695] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.427592 21520 leader_election.cc:304] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.428637 21594 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.414386 24564 raft_consensus.cc:2798] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:43.408857 24322 raft_consensus.cc:2743] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 08:59:43.433178 21695 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 10. Current term is 11. Ops: []
I20250114 08:59:43.416227 24590 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:43.437219 21624 raft_consensus.cc:1270] T 3b369b35736c413ca069db061ed36bc5 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250114 08:59:43.416991 24184 raft_consensus.cc:2743] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:43.431177 21703 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:43.443070 21703 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:43.445254 24676 log.cc:926] Time spent T 89ff7b2c5f4e465fb847cd8b1c0b9e5a P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20250114 08:59:43.445016 24596 consensus_queue.cc:1035] T 3b369b35736c413ca069db061ed36bc5 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250114 08:59:43.446846 22266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49020 (request call id 319) took 2248 ms. Trace:
I20250114 08:59:43.447229 22266 rpcz_store.cc:276] 0114 08:59:41.198728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:41.199320 (+ 592us) service_pool.cc:225] Handling call
0114 08:59:42.709290 (+1509970us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b5800003698. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63c4d7911 00007fa63c656991 00007fa63c656842 00007fa645327cd9 00007fa6444b6750 00007fa64457c7fb 00007fa64457c881 00007fa6445402a0 00007fa64452b7e9 00007fa64452b292 00007fa63dbf7694
0114 08:59:42.749260 (+ 39970us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 08:59:42.784806 (+ 35546us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 08:59:42.790231 (+ 5425us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 08:59:42.870225 (+ 79994us) log.cc:844] Serialized 26 byte log entry
0114 08:59:43.064653 (+194428us) raft_consensus.cc:1640] Marking committed up to 6
0114 08:59:43.068693 (+ 4040us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:43.076768 (+ 8075us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 08:59:43.446505 (+369737us) raft_consensus.cc:1680] finished
0114 08:59:43.446533 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 08:59:43.446759 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13708544,"thread_start_us":119887,"threads_started":1,"wal-append.queue_time_us":340559}
I20250114 08:59:43.449364 24654 consensus_queue.cc:237] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.452709 24395 consensus_queue.cc:237] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.456789 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3650a91ff4a4213b9091f197bf75fb9" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
W20250114 08:59:43.457307 24678 log.cc:926] Time spent T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20250114 08:59:43.457568 21697 raft_consensus.cc:3054] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 3 FOLLOWER]: Advancing to term 4
I20250114 08:59:43.426388 24630 raft_consensus.cc:738] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.459548 24630 consensus_queue.cc:260] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.460836 24630 raft_consensus.cc:3054] T 6e4913bb74ad4fab98dd737eb0bd5965 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:43.426153 24604 raft_consensus.cc:3054] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:43.471132 21697 raft_consensus.cc:2463] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 4.
I20250114 08:59:43.430311 24607 raft_consensus.cc:2743] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:43.427511 24468 raft_consensus.cc:695] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.492870 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:43.493327 24468 consensus_queue.cc:237] T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.433954 24370 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:43.496584 24567 raft_consensus.cc:2743] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:43.496805 24680 log.cc:926] Time spent T addec468f1bd4f19a98a8d5ce1222fc1 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
W20250114 08:59:43.498086 24684 log.cc:926] Time spent T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250114 08:59:43.431056 24549 raft_consensus.cc:695] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:43.506534 21518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 55627.6us
W20250114 08:59:43.509183 21520 connection.cc:232] RPC call timeout handler was delayed by -1.0395350309999003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
I20250114 08:59:43.423739 24279 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20250114 08:59:43.511183 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93e2235a71d41a4a46a1e50fd8c02b7" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:43.513494 21703 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.442340 24590 raft_consensus.cc:491] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:43.519848 21703 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:43.462853 24564 raft_consensus.cc:695] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.513947 21700 raft_consensus.cc:2371] T a93e2235a71d41a4a46a1e50fd8c02b7 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9. Re-sending same reply.
W20250114 08:59:43.519454 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 76321.4us
I20250114 08:59:43.501189 24656 raft_consensus.cc:2798] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:43.496913 24370 raft_consensus.cc:491] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:43.529112 21521 leader_election.cc:304] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters:
W20250114 08:59:43.533717 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 105142us
I20250114 08:59:43.534488 21594 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.522141 24590 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:43.555115 24671 raft_consensus.cc:2798] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:43.535497 24370 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:43.539491 22266 raft_consensus.cc:1235] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:43.576071 21551 raft_consensus.cc:1235] T 8aa5b3173173431d91e459f9fdcb61a9 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:43.530755 24530 consensus_queue.cc:1035] T ee02341e5e574d55aac7eb9b3df9dc73 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20250114 08:59:43.596580 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:43.489822 24334 consensus_queue.cc:260] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.600972 24334 raft_consensus.cc:3054] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:43.616942 24564 consensus_queue.cc:237] T 9690a0a2cde84c5d9312c8694185a0b2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.621971 21629 raft_consensus.cc:1270] T 2e77833a006f478e86764142c505ab6b P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250114 08:59:43.568848 24687 raft_consensus.cc:491] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:43.578843 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:43.581018 24670 leader_election.cc:290] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:43.552493 24694 raft_consensus.cc:491] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:43.583009 24701 consensus_queue.cc:1046] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250114 08:59:43.628615 21628 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:43.542820 24656 raft_consensus.cc:695] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:43.662482 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 136878us
I20250114 08:59:43.565687 24547 raft_consensus.cc:2798] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:43.577003 24619 leader_election.cc:290] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:43.712415 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 126714us
W20250114 08:59:43.676793 24657 consensus_meta.cc:287] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.510s user 0.015s sys 0.000s
W20250114 08:59:43.756563 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 184945us
I20250114 08:59:43.748436 24366 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:43.670362 24596 consensus_queue.cc:1035] T 2e77833a006f478e86764142c505ab6b P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20250114 08:59:43.769809 21520 leader_election.cc:304] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:43.767119 24655 consensus_queue.cc:1046] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250114 08:59:43.575274 24549 consensus_queue.cc:237] T b2d157d78e1e4854b971c0382cb7fe21 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.772054 24671 raft_consensus.cc:695] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:43.772683 21628 raft_consensus.cc:1270] T 426decde999b4722a874569acae5664f P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250114 08:59:43.793820 24547 raft_consensus.cc:695] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [term 7 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:43.795166 24547 consensus_queue.cc:237] T 7050054fd28749bca69cd4c2e1336d7b P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:43.755327 24619 consensus_queue.cc:1035] T baef410f99d34fe9a53080abafb2259e P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
W20250114 08:59:43.814366 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 44704.5us
I20250114 08:59:43.663810 24694 raft_consensus.cc:513] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.817838 21520 leader_election.cc:304] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.670121 24687 raft_consensus.cc:513] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } attrs { replace: true } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:43.817193 24279 consensus_queue.cc:1035] T 426decde999b4722a874569acae5664f P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20250114 08:59:43.814239 24530 raft_consensus.cc:2758] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250114 08:59:43.857298 21520 leader_election.cc:304] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:43.778057 24366 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
W20250114 08:59:43.887476 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 210390us
I20250114 08:59:43.859488 24713 raft_consensus.cc:2743] T abedb4afaad04af9bbbb7d4266fd2632 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:43.932937 24704 log.cc:926] Time spent T 0205fa1ad333433a8a509e5ad9620de8 P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20250114 08:59:43.939435 24590 raft_consensus.cc:513] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.786861 24656 consensus_queue.cc:237] T a5d394e52b4f4ee2be4e4f4a32ce8547 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:43.817135 24712 consensus_queue.cc:1035] T b2d157d78e1e4854b971c0382cb7fe21 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20250114 08:59:43.859846 24347 raft_consensus.cc:2743] T d0b4ce684a6943809542800e14d94079 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250114 08:59:43.961732 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 72508.9us
I20250114 08:59:43.954070 24686 raft_consensus.cc:3049] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 LEADER]: Stepping down as leader of term 10
I20250114 08:59:43.825321 24671 consensus_queue.cc:237] T a93e2235a71d41a4a46a1e50fd8c02b7 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:43.985193 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 126683us
I20250114 08:59:43.987897 21669 leader_election.cc:304] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:43.991884 21669 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.005769 21669 leader_election.cc:304] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:44.006004 21520 leader_election.cc:304] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.007269 21520 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.007378 21669 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.008684 21520 leader_election.cc:400] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 5 pre-election: Vote denied by peer d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891) with higher term. Message: Invalid argument: T 1b5c7ffbe79e4145825848346fd7652c P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6.
I20250114 08:59:44.009805 24366 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:44.010181 21520 leader_election.cc:304] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.010371 24366 raft_consensus.cc:2743] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 08:59:44.011307 21520 leader_election.cc:304] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:44.011373 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:44.012576 22269 raft_consensus.cc:2388] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28a83b7a7d9d4aa899b425086aa45a8e in current term 9: Already voted for candidate 57f9cc1dd3c64647848a33a65b08c893 in this term.
I20250114 08:59:44.012640 21520 leader_election.cc:304] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:44.013935 24395 raft_consensus.cc:2798] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 08:59:44.014457 24395 raft_consensus.cc:491] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:44.014927 24395 raft_consensus.cc:3054] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:44.015682 24567 raft_consensus.cc:2758] T ada199404e61479593fa1b64c574bcb8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250114 08:59:43.986617 24686 raft_consensus.cc:738] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:44.014935 21520 leader_election.cc:304] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:43.998642 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 139706us
I20250114 08:59:44.021368 24315 raft_consensus.cc:2798] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 08:59:44.041518 24315 raft_consensus.cc:491] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.042119 24315 raft_consensus.cc:3054] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Advancing to term 6
W20250114 08:59:44.043938 24390 consensus_meta.cc:287] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893: Time spent flushing consensus metadata: real 0.601s user 0.006s sys 0.003s
I20250114 08:59:44.044417 24390 raft_consensus.cc:513] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:44.047084 22273 raft_consensus.cc:1235] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:44.047415 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5506492f514407a5097b5b2e8c7042" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:44.049296 24390 leader_election.cc:290] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:44.024575 24694 leader_election.cc:290] T cc5506492f514407a5097b5b2e8c7042 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:44.023890 24030 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250114 08:59:44.017235 24669 raft_consensus.cc:2743] T 270ce1b254bd4b7180e48e18af79cda6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 08:59:44.019428 24366 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:44.056531 24590 leader_election.cc:290] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.057672 21520 leader_election.cc:304] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:44.058810 21695 raft_consensus.cc:1235] T 8e6414eaae2e412fa959acd24b4f7368 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:44.063293 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.067140 24184 raft_consensus.cc:2798] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:44.067530 21520 leader_election.cc:304] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:44.067795 24184 raft_consensus.cc:491] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
I20250114 08:59:44.019150 24699 raft_consensus.cc:2798] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 08:59:44.069854 24699 raft_consensus.cc:695] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:44.070747 24366 raft_consensus.cc:2743] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:44.071118 24699 consensus_queue.cc:237] T a1aeae7c29264f839189001364c51746 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.073930 24669 raft_consensus.cc:2798] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 08:59:44.074470 24669 raft_consensus.cc:491] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting leader election (detected failure of leader cec0ef3bbb654988bf2123d8eef7a0a7)
W20250114 08:59:44.074849 24604 consensus_meta.cc:287] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.574s user 0.009s sys 0.000s
I20250114 08:59:44.075017 24669 raft_consensus.cc:3054] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:44.075351 24604 raft_consensus.cc:2743] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250114 08:59:44.078868 24581 consensus_meta.cc:287] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.705s user 0.008s sys 0.006s
I20250114 08:59:44.079382 24581 raft_consensus.cc:513] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:44.082221 24370 raft_consensus.cc:513] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.087630 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.090420 21697 raft_consensus.cc:1270] T 001d4f9f4b5d4124b95dcddb6adac2d6 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250114 08:59:44.090984 21547 raft_consensus.cc:1235] T b7929f46385a41b0acb9e10a87605e22 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Rejecting Update request from peer 57f9cc1dd3c64647848a33a65b08c893 for earlier term 10. Current term is 11. Ops: []
I20250114 08:59:44.091562 21520 leader_election.cc:304] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:44.025087 21669 leader_election.cc:304] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
W20250114 08:59:44.100555 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 76125.6us
I20250114 08:59:44.109366 21520 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.068351 24184 raft_consensus.cc:3054] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
W20250114 08:59:44.173924 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 62080.5us
W20250114 08:59:44.058851 24653 consensus_meta.cc:287] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.771s user 0.011s sys 0.000s
I20250114 08:59:43.997912 24715 raft_consensus.cc:2798] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250114 08:59:44.189519 24715 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.189993 24715 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:44.106621 21669 leader_election.cc:304] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.017746 24713 consensus_queue.cc:1035] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.174s
W20250114 08:59:44.193217 24334 consensus_meta.cc:287] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed: Time spent flushing consensus metadata: real 0.592s user 0.012s sys 0.000s
I20250114 08:59:44.174213 24567 raft_consensus.cc:2743] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:44.194084 21626 raft_consensus.cc:1270] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Refusing update from remote peer 28a83b7a7d9d4aa899b425086aa45a8e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250114 08:59:44.191026 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 84520.3us
I20250114 08:59:44.195108 24673 consensus_queue.cc:1046] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 08:59:44.195433 21669 leader_election.cc:304] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:44.196065 24673 raft_consensus.cc:3049] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:44.196439 24673 raft_consensus.cc:738] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:44.196645 24712 raft_consensus.cc:2798] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:44.196890 21669 leader_election.cc:304] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:44.197362 24712 raft_consensus.cc:695] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:44.197265 24673 consensus_queue.cc:260] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.198410 24643 raft_consensus.cc:2798] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 08:59:44.198807 24673 raft_consensus.cc:3054] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:44.198839 24643 raft_consensus.cc:491] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.199532 24643 raft_consensus.cc:3054] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Advancing to term 5
I20250114 08:59:44.198686 24712 consensus_queue.cc:237] T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.200856 24451 consensus_queue.cc:1046] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:59:44.202245 24597 raft_consensus.cc:2758] T 3b369b35736c413ca069db061ed36bc5 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250114 08:59:44.203101 24597 consensus_queue.cc:1046] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:59:44.195806 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:44.204946 22267 raft_consensus.cc:3054] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:44.023658 24707 raft_consensus.cc:2743] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250114 08:59:44.208227 24707 raft_consensus.cc:3049] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:44.208665 24707 raft_consensus.cc:738] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:44.209656 24707 consensus_queue.cc:260] T 7050054fd28749bca69cd4c2e1336d7b P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } }
I20250114 08:59:44.212101 24606 raft_consensus.cc:988] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e: : Instructing follower cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:44.212762 24606 raft_consensus.cc:1076] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Signalling peer cec0ef3bbb654988bf2123d8eef7a0a7 to start an election
I20250114 08:59:44.173738 24694 raft_consensus.cc:2798] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250114 08:59:44.214776 24694 raft_consensus.cc:491] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.215124 24694 raft_consensus.cc:3054] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:44.219111 24397 consensus_queue.cc:1035] T 001d4f9f4b5d4124b95dcddb6adac2d6 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250114 08:59:44.094720 24376 consensus_meta.cc:287] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7: Time spent flushing consensus metadata: real 0.671s user 0.009s sys 0.004s
I20250114 08:59:44.184815 24581 leader_election.cc:290] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:44.084398 24370 leader_election.cc:290] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
W20250114 08:59:44.233031 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 56703.3us
I20250114 08:59:44.285233 21520 leader_election.cc:304] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, cec0ef3bbb654988bf2123d8eef7a0a7; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.170732 24279 raft_consensus.cc:491] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:44.284047 21699 raft_consensus.cc:2463] T 6846f3a6d80748369341fe169ebc427e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a83b7a7d9d4aa899b425086aa45a8e in term 9.
I20250114 08:59:44.189811 22269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893" is_pre_election: true
I20250114 08:59:44.172981 24030 raft_consensus.cc:2758] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250114 08:59:44.288270 21697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50"
dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
from {username='slave'} at 127.0.0.1:46398
I20250114 08:59:44.174974 24687 leader_election.cc:290] T 82452b55145444529bf0609cd7767adf P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:44.294201 21520 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
W20250114 08:59:44.283038 24654 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.110s sys 0.158s
I20250114 08:59:44.297488 24730 raft_consensus.cc:2743] T 82452b55145444529bf0609cd7767adf P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250114 08:59:44.087656 24686 consensus_queue.cc:260] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.300798 24686 raft_consensus.cc:3054] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Advancing to term 11
I20250114 08:59:44.204254 24395 raft_consensus.cc:513] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.293061 24279 raft_consensus.cc:513] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:44.306255 24395 leader_election.cc:290] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.306427 24279 leader_election.cc:290] T 6846f3a6d80748369341fe169ebc427e P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.309055 22267 raft_consensus.cc:2463] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 9.
I20250114 08:59:44.310832 21593 leader_election.cc:304] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.320986 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb0c7bd28441aabcf6344f04b84ee6" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:44.319059 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6846f3a6d80748369341fe169ebc427e" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.334272 24030 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:44.334858 24030 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Leader. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:44.335712 24030 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:44.333760 21520 leader_election.cc:304] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:44.344616 24604 raft_consensus.cc:2798] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:44.345064 24604 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:44.345396 24604 raft_consensus.cc:3054] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:44.344409 24315 raft_consensus.cc:513] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.346746 21696 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:44.349565 21520 leader_election.cc:304] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.352948 21520 leader_election.cc:304] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.355007 21520 leader_election.cc:304] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:44.298110 24654 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s user 0.110s sys 0.158s
I20250114 08:59:44.352563 24315 leader_election.cc:290] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:44.353696 24395 raft_consensus.cc:2743] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 08:59:44.383400 21520 leader_election.cc:304] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.385783 21520 leader_election.cc:304] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed
W20250114 08:59:44.472605 21668 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.014s sys 0.020s
W20250114 08:59:44.407415 24695 log.cc:926] Time spent T 92fb0c7bd28441aabcf6344f04b84ee6 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.957s user 0.004s sys 0.014s
I20250114 08:59:44.356513 24715 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.360141 24184 raft_consensus.cc:513] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:44.473263 21668 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.515s user 0.014s sys 0.021s
W20250114 08:59:44.473686 21668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 515771us
W20250114 08:59:44.475003 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 89326.1us
I20250114 08:59:44.478929 21520 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.386667 24606 raft_consensus.cc:3054] T a03f2f2f70724b85b8b23ed7bc72fbe1 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:44.386873 24366 raft_consensus.cc:2743] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:44.286120 24669 raft_consensus.cc:513] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.361617 24607 raft_consensus.cc:2743] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 08:59:44.481644 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 283534us
I20250114 08:59:44.492283 24184 leader_election.cc:290] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.493670 21520 leader_election.cc:304] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.494915 24694 raft_consensus.cc:513] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.496353 24687 consensus_queue.cc:1035] T 106dea9d52554552a1b3e20e7c82a3a8 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:59:44.496899 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80026dc5df8c4456a48f734e1a64b930" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:44.497848 24643 raft_consensus.cc:513] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.498096 24687 consensus_queue.cc:1035] T 1b5c7ffbe79e4145825848346fd7652c P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 08:59:44.499912 24643 leader_election.cc:290] T be5a427c5ff04f9fa27e3e64bd27b146 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 5 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:44.488211 24709 raft_consensus.cc:2743] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250114 08:59:44.504272 24699 consensus_queue.cc:1046] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250114 08:59:44.505764 24687 raft_consensus.cc:3049] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:44.506125 24687 raft_consensus.cc:738] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:44.506554 24669 leader_election.cc:290] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.506886 24687 consensus_queue.cc:260] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.507923 24687 raft_consensus.cc:3054] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:44.509882 24694 leader_election.cc:290] T 6e6879e06c934d438e3ca9c3f093d4be P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 11 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:44.510725 24720 raft_consensus.cc:491] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.511193 24643 consensus_queue.cc:1035] T 4c244f79365347c49406af3a7a5a55a7 P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 08:59:44.511276 24720 raft_consensus.cc:513] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
W20250114 08:59:44.481151 24685 log.cc:926] Time spent T a14dbff33afe4ab483f01265cb35d676 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250114 08:59:44.517853 21700 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
W20250114 08:59:44.538837 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 45239.7us
I20250114 08:59:44.544960 21520 leader_election.cc:304] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters: cec0ef3bbb654988bf2123d8eef7a0a7
I20250114 08:59:44.483180 24395 raft_consensus.cc:2743] T 6e4913bb74ad4fab98dd737eb0bd5965 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250114 08:59:44.578974 21555 raft_consensus.cc:1270] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer d7732324fb5f48d3b16a29d4bb2584ed: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 08:59:44.581929 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:44.582559 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fa6dec61d44618fee387c9b1f2994" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.517045 24564 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:44.608328 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa184c9a07ba40518b2e4d61a40e4da2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.609306 21551 raft_consensus.cc:3054] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:44.611073 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4174b5896143c7b50c7abfd618a899" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.611919 21548 raft_consensus.cc:3054] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
W20250114 08:59:44.614306 21520 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 69494us
I20250114 08:59:44.615149 21520 leader_election.cc:304] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e; no voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed
I20250114 08:59:44.517266 24630 raft_consensus.cc:491] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.620199 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.621730 21554 raft_consensus.cc:3054] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:44.622833 21549 raft_consensus.cc:1235] T a5d394e52b4f4ee2be4e4f4a32ce8547 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:44.581774 24713 raft_consensus.cc:2743] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 08:59:44.634022 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.642088 21551 raft_consensus.cc:2463] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 10.
I20250114 08:59:44.648033 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.624408 24395 raft_consensus.cc:2743] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 08:59:44.687067 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.699997 21549 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:44.700845 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e70c3cb3e8e4656945ec18e88e46101" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.673607 20382 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 08:59:44.702107 21549 raft_consensus.cc:1235] T 0205fa1ad333433a8a509e5ad9620de8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:44.621619 24366 raft_consensus.cc:2798] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250114 08:59:44.683074 21593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.132:35045, user_credentials={real_user=slave}} blocked reactor thread for 372300us
W20250114 08:59:44.684963 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 104086us
I20250114 08:59:44.614677 24184 consensus_queue.cc:1046] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20250114 08:59:44.604599 24322 consensus_queue.cc:1046] T f21ba5689065467b967a5802582959dd P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20250114 08:59:44.745922 21550 raft_consensus.cc:1235] T b2d157d78e1e4854b971c0382cb7fe21 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 08:59:44.689646 21548 raft_consensus.cc:2463] T cc4174b5896143c7b50c7abfd618a899 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:44.676882 24030 consensus_queue.cc:1046] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20250114 08:59:44.633302 24630 raft_consensus.cc:513] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.697643 24715 leader_election.cc:290] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
W20250114 08:59:44.714010 21669 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 218436us
W20250114 08:59:44.659683 21596 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.231.131:37485 timed out after 1.545s (SENT) [suppressed 73 similar messages]
I20250114 08:59:44.715103 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.688822 21546 raft_consensus.cc:1235] T fa184c9a07ba40518b2e4d61a40e4da2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20250114 08:59:44.756971 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.757761 21555 raft_consensus.cc:3054] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:44.689154 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.758934 21669 leader_election.cc:304] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.685945 24607 raft_consensus.cc:491] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:44.683673 24657 raft_consensus.cc:2798] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Leader election won for term 9
I20250114 08:59:44.749445 21548 raft_consensus.cc:1235] T 6c3a3fdbbbca423695a05eb175031e72 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 10. Current term is 11. Ops: []
I20250114 08:59:44.767094 21548 raft_consensus.cc:1235] T 7050054fd28749bca69cd4c2e1336d7b P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:44.769513 21548 raft_consensus.cc:1270] T 1b5c7ffbe79e4145825848346fd7652c P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 08:59:44.614902 24564 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.772004 24564 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.753463 21594 leader_election.cc:304] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.755131 21697 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 08:59:44.773530 21697 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:44.715680 24366 raft_consensus.cc:491] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 08:59:44.746434 24723 raft_consensus.cc:3049] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:44.789819 21548 raft_consensus.cc:1235] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:44.800295 21669 leader_election.cc:304] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:44.803694 21669 leader_election.cc:304] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 57f9cc1dd3c64647848a33a65b08c893
I20250114 08:59:44.804879 21548 raft_consensus.cc:1270] T 4c244f79365347c49406af3a7a5a55a7 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Refusing update from remote peer cec0ef3bbb654988bf2123d8eef7a0a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20250114 08:59:44.791535 24702 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.010s sys 0.038s
W20250114 08:59:44.822088 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 68697.6us
I20250114 08:59:44.834708 21594 leader_election.cc:304] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.722309 24370 consensus_queue.cc:1035] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20250114 08:59:44.756153 24655 raft_consensus.cc:491] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:44.799551 24366 raft_consensus.cc:3054] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:44.838814 21555 raft_consensus.cc:2463] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:44.839313 21697 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.845984 21555 raft_consensus.cc:1235] T 9690a0a2cde84c5d9312c8694185a0b2 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:44.837267 24564 raft_consensus.cc:2798] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 FOLLOWER]: Leader election won for term 10
I20250114 08:59:44.800401 24723 raft_consensus.cc:738] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
I20250114 08:59:44.804095 24720 leader_election.cc:290] T 82151d61cef04050b69f24bc614507fc P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:44.836711 24657 raft_consensus.cc:695] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
W20250114 08:59:44.853580 21596 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 193922us
I20250114 08:59:44.864832 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21ba5689065467b967a5802582959dd" candidate_uuid: "57f9cc1dd3c64647848a33a65b08c893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7732324fb5f48d3b16a29d4bb2584ed"
I20250114 08:59:44.865489 21624 raft_consensus.cc:3054] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 5 FOLLOWER]: Advancing to term 6
I20250114 08:59:44.855867 21554 raft_consensus.cc:2463] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8.
I20250114 08:59:44.869253 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82452b55145444529bf0609cd7767adf" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.870798 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.859380 21551 raft_consensus.cc:1235] T 80026dc5df8c4456a48f734e1a64b930 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 8. Current term is 9. Ops: []
W20250114 08:59:44.833801 24702 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.997s user 0.010s sys 0.038s
W20250114 08:59:44.861328 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 93688.4us
I20250114 08:59:44.823268 24739 raft_consensus.cc:2758] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250114 08:59:44.810241 24243 raft_consensus.cc:2798] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250114 08:59:44.844807 21669 leader_election.cc:304] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0ef3bbb654988bf2123d8eef7a0a7, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:44.863972 21697 leader_election.cc:290] T fec3bfdda69d4934970f1e752db88d50 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 7 election: Requested vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:44.879091 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.880808 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.882350 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7929f46385a41b0acb9e10a87605e22" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.886260 21697 raft_consensus.cc:1235] T b3650a91ff4a4213b9091f197bf75fb9 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:44.889741 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5a427c5ff04f9fa27e3e64bd27b146" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.890909 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6414eaae2e412fa959acd24b4f7368" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
W20250114 08:59:44.892946 22239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 86969.2us
I20250114 08:59:44.893328 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7e8860a1d145fbb31ef3dc7864f2d0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.894016 21555 raft_consensus.cc:2463] T 4c7e8860a1d145fbb31ef3dc7864f2d0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9.
I20250114 08:59:44.845455 24746 raft_consensus.cc:2798] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250114 08:59:44.876909 24741 log.cc:926] Time spent T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250114 08:59:44.896392 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7050054fd28749bca69cd4c2e1336d7b" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.790421 24607 raft_consensus.cc:513] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.896207 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.876787 24334 raft_consensus.cc:2798] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:44.900794 21624 raft_consensus.cc:2463] T f21ba5689065467b967a5802582959dd P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f9cc1dd3c64647848a33a65b08c893 in term 6.
I20250114 08:59:44.899175 21555 raft_consensus.cc:1235] T 92fb0c7bd28441aabcf6344f04b84ee6 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 7. Current term is 8. Ops: []
I20250114 08:59:44.874636 24243 raft_consensus.cc:491] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:44.907586 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6879e06c934d438e3ca9c3f093d4be" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.912285 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8afddecbe8465193b684ed870589f0" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:44.912974 21697 raft_consensus.cc:1235] T 3e70c3cb3e8e4656945ec18e88e46101 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:44.913178 21555 raft_consensus.cc:2405] T ec8afddecbe8465193b684ed870589f0 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20250114 08:59:44.873657 24720 raft_consensus.cc:491] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:44.908602 21554 raft_consensus.cc:2463] T 8e6414eaae2e412fa959acd24b4f7368 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:44.890188 24715 raft_consensus.cc:2798] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 08:59:44.916138 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0564bed845d3430e8831ee6428122144" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.915064 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d394e52b4f4ee2be4e4f4a32ce8547" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.918231 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f2f2f70724b85b8b23ed7bc72fbe1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.927500 21549 raft_consensus.cc:2463] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 6.
I20250114 08:59:44.929915 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a451ff14593400ca08caa1b5ed169f2" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.933948 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0205fa1ad333433a8a509e5ad9620de8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.851686 24655 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.896931 24746 raft_consensus.cc:491] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57f9cc1dd3c64647848a33a65b08c893)
I20250114 08:59:44.940924 21551 raft_consensus.cc:1235] T 0564bed845d3430e8831ee6428122144 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 6. Current term is 7. Ops: []
I20250114 08:59:44.912201 24243 raft_consensus.cc:3054] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Advancing to term 10
I20250114 08:59:44.950771 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d157d78e1e4854b971c0382cb7fe21" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:44.961923 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 80685.6us
I20250114 08:59:44.961802 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270ce1b254bd4b7180e48e18af79cda6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.915848 24334 raft_consensus.cc:695] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:44.984161 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:44.987169 21551 raft_consensus.cc:2388] T a1aeae7c29264f839189001364c51746 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cec0ef3bbb654988bf2123d8eef7a0a7 in current term 7: Already voted for candidate 28a83b7a7d9d4aa899b425086aa45a8e in this term.
I20250114 08:59:44.990285 22240 leader_election.cc:304] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57f9cc1dd3c64647848a33a65b08c893, d7732324fb5f48d3b16a29d4bb2584ed; no voters: 28a83b7a7d9d4aa899b425086aa45a8e
I20250114 08:59:44.991374 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e648ebc4e4d4a4b9a35eefeb989e0fe" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:44.945566 24715 raft_consensus.cc:695] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 LEADER]: Becoming Leader. State: Replica: cec0ef3bbb654988bf2123d8eef7a0a7, State: Running, Role: LEADER
W20250114 08:59:44.962200 24732 log.cc:926] Time spent T 3e648ebc4e4d4a4b9a35eefeb989e0fe P d7732324fb5f48d3b16a29d4bb2584ed: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20250114 08:59:44.945811 24746 raft_consensus.cc:3054] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Advancing to term 8
I20250114 08:59:44.998435 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001d4f9f4b5d4124b95dcddb6adac2d6" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:44.915827 24723 consensus_queue.cc:260] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.999603 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.000870 24756 raft_consensus.cc:2798] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 08:59:44.939781 24720 raft_consensus.cc:513] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.956094 24657 consensus_queue.cc:237] T d19fa6dec61d44618fee387c9b1f2994 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:45.005506 21547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50144 (request call id 229) took 1561 ms. Trace:
I20250114 08:59:45.031697 21547 rpcz_store.cc:276] 0114 08:59:43.444005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:43.444158 (+ 153us) service_pool.cc:225] Handling call
0114 08:59:43.444597 (+ 439us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:44.091407 (+646810us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:45.005220 (+913813us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b540008d254. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:45.005415 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1486101888}
W20250114 08:59:45.005645 24745 log.cc:926] Time spent T 073e9192e9cb48728d4dd8027da50f60 P cec0ef3bbb654988bf2123d8eef7a0a7: Append to log took a long time: real 0.163s user 0.001s sys 0.001s
I20250114 08:59:45.006906 24334 consensus_queue.cc:237] T cc4174b5896143c7b50c7abfd618a899 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:44.995290 22273 raft_consensus.cc:1235] T b3650a91ff4a4213b9091f197bf75fb9 P 57f9cc1dd3c64647848a33a65b08c893 [term 4 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 3. Current term is 4. Ops: []
I20250114 08:59:45.009902 22267 raft_consensus.cc:1235] T d19fa6dec61d44618fee387c9b1f2994 P 57f9cc1dd3c64647848a33a65b08c893 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
W20250114 08:59:44.991971 24744 log.cc:926] Time spent T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20250114 08:59:44.992548 24757 raft_consensus.cc:491] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:45.047964 24757 raft_consensus.cc:513] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.040906 24756 raft_consensus.cc:695] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [term 6 LEADER]: Becoming Leader. State: Replica: 57f9cc1dd3c64647848a33a65b08c893, State: Running, Role: LEADER
I20250114 08:59:45.048812 24746 raft_consensus.cc:513] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:45.049459 24756 consensus_queue.cc:237] T f21ba5689065467b967a5802582959dd P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:45.050215 24757 leader_election.cc:290] T 3ba4257b18434159a50a4f6ae4a0d736 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:45.051160 24746 leader_election.cc:290] T cc9f4dcde58e4d508d5ffbbf301a7a6b P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 8 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), 57f9cc1dd3c64647848a33a65b08c893 (127.19.231.132:35045)
I20250114 08:59:45.051600 24709 raft_consensus.cc:3049] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:45.052004 24709 raft_consensus.cc:738] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:45.053174 24687 raft_consensus.cc:491] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:45.052820 24709 consensus_queue.cc:260] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.053310 21621 raft_consensus.cc:1270] T ada199404e61479593fa1b64c574bcb8 P d7732324fb5f48d3b16a29d4bb2584ed [term 7 FOLLOWER]: Refusing update from remote peer 57f9cc1dd3c64647848a33a65b08c893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250114 08:59:45.053856 24709 raft_consensus.cc:3054] T 61f37f8d5b9d4c71b6e852f77fa438e8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Advancing to term 9
I20250114 08:59:45.053699 24687 raft_consensus.cc:513] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.055698 24687 leader_election.cc:290] T e9e2d1d6b0f14582b7864c1b80a47f26 P cec0ef3bbb654988bf2123d8eef7a0a7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891)
I20250114 08:59:45.056524 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106dea9d52554552a1b3e20e7c82a3a8" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.057166 24716 consensus_queue.cc:1046] T b7929f46385a41b0acb9e10a87605e22 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250114 08:59:44.998243 24630 leader_election.cc:290] T 8e6414eaae2e412fa959acd24b4f7368 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:45.037966 21696 raft_consensus.cc:1235] T 61f37f8d5b9d4c71b6e852f77fa438e8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 8. Current term is 9. Ops: []
I20250114 08:59:45.058610 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d803a1f3141467fa3c2c63696526f84" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.062975 21702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addec468f1bd4f19a98a8d5ce1222fc1" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.060007 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff7b2c5f4e465fb847cd8b1c0b9e5a" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.047148 24720 leader_election.cc:290] T 8aa5b3173173431d91e459f9fdcb61a9 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:45.057706 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.064764 21697 raft_consensus.cc:2363] T 125d011b631240b8bd1138f8cd52ec2e P cec0ef3bbb654988bf2123d8eef7a0a7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for earlier term 8. Current term is 9.
I20250114 08:59:44.989791 24564 raft_consensus.cc:695] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [term 10 LEADER]: Becoming Leader. State: Replica: d7732324fb5f48d3b16a29d4bb2584ed, State: Running, Role: LEADER
I20250114 08:59:45.068429 21701 raft_consensus.cc:1235] T 426decde999b4722a874569acae5664f P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:45.068660 22266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f4dcde58e4d508d5ffbbf301a7a6b" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57f9cc1dd3c64647848a33a65b08c893"
I20250114 08:59:45.031155 24702 raft_consensus.cc:491] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 28a83b7a7d9d4aa899b425086aa45a8e)
I20250114 08:59:45.062736 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb00de7db5024821adbfdfd3f8a0f475" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:45.072577 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.073654 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.074515 24366 raft_consensus.cc:513] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.081342 24655 leader_election.cc:290] T baef410f99d34fe9a53080abafb2259e P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 28a83b7a7d9d4aa899b425086aa45a8e (127.19.231.129:45539), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:45.082413 24564 consensus_queue.cc:237] T fa184c9a07ba40518b2e4d61a40e4da2 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.083052 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52f2d242e445319680653bc6c30fff" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:45.083664 21551 raft_consensus.cc:2371] T 5b52f2d242e445319680653bc6c30fff P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 8. Re-sending same reply.
I20250114 08:59:45.040908 24723 raft_consensus.cc:3054] T 3e648ebc4e4d4a4b9a35eefeb989e0fe P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Advancing to term 7
I20250114 08:59:45.057871 24728 raft_consensus.cc:491] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:45.089385 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeae7c29264f839189001364c51746" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.070063 24687 raft_consensus.cc:491] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:45.106717 21548 raft_consensus.cc:1235] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10. Ops: []
I20250114 08:59:45.109562 21703 raft_consensus.cc:1235] T 3ba4257b18434159a50a4f6ae4a0d736 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Rejecting Update request from peer d7732324fb5f48d3b16a29d4bb2584ed for earlier term 10. Current term is 11. Ops: []
W20250114 08:59:45.116091 22240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.130:36891, user_credentials={real_user=slave}} blocked reactor thread for 74152.6us
I20250114 08:59:45.114578 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.125427 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125d011b631240b8bd1138f8cd52ec2e" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:45.126060 21551 raft_consensus.cc:2371] T 125d011b631240b8bd1138f8cd52ec2e P 28a83b7a7d9d4aa899b425086aa45a8e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 9. Re-sending same reply.
W20250114 08:59:45.106554 24604 consensus_meta.cc:287] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e: Time spent flushing consensus metadata: real 0.761s user 0.011s sys 0.002s
I20250114 08:59:45.128613 21696 raft_consensus.cc:2463] T eb00de7db5024821adbfdfd3f8a0f475 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 7.
I20250114 08:59:45.113596 24607 leader_election.cc:290] T 073e9192e9cb48728d4dd8027da50f60 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:45.136240 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.079368 24243 raft_consensus.cc:513] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.141281 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c244f79365347c49406af3a7a5a55a7" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
I20250114 08:59:45.155172 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f37f8d5b9d4c71b6e852f77fa438e8" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:45.161175 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e9192e9cb48728d4dd8027da50f60" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:45.162586 21696 raft_consensus.cc:1235] T ee02341e5e574d55aac7eb9b3df9dc73 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 10 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 9. Current term is 10. Ops: []
I20250114 08:59:45.165868 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3a3fdbbbca423695a05eb175031e72" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:45.166677 21703 raft_consensus.cc:2405] T 6c3a3fdbbbca423695a05eb175031e72 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d7732324fb5f48d3b16a29d4bb2584ed for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20250114 08:59:45.128221 24184 consensus_queue.cc:1046] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20250114 08:59:45.086814 24715 consensus_queue.cc:237] T 5b52f2d242e445319680653bc6c30fff P cec0ef3bbb654988bf2123d8eef7a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.171325 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14dbff33afe4ab483f01265cb35d676" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7"
I20250114 08:59:45.110284 24687 raft_consensus.cc:513] T ada199404e61479593fa1b64c574bcb8 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } attrs { replace: true } } peers { permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false } }
I20250114 08:59:45.172161 21697 raft_consensus.cc:1235] T 6a451ff14593400ca08caa1b5ed169f2 P cec0ef3bbb654988bf2123d8eef7a0a7 [term 6 FOLLOWER]: Rejecting Update request from peer 28a83b7a7d9d4aa899b425086aa45a8e for earlier term 5. Current term is 6. Ops: []
I20250114 08:59:45.177472 21626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57318 (request call id 1806) took 1958 ms. Trace:
I20250114 08:59:45.177866 21626 rpcz_store.cc:276] 0114 08:59:43.218940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 08:59:43.354177 (+135237us) service_pool.cc:225] Handling call
0114 08:59:43.373061 (+ 18884us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 08:59:44.194569 (+821508us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 08:59:45.177241 (+982672us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b54000c9c54. stack: 00007fa63c661f63 00007fa63c661e60 00007fa63bfb5584 00007fa645aa9ff4 00007fa645aa9f89 00007fa63f5526d1 00007fa63f5cdf51 00007fa63f5cc99e 00007fa64452b2d9 00007fa63dbf7694 00007fa63dbf7622 00007fa63dbf7551 00007fa63dbf74cc 00007fa63dbf6782 00007fa63d9f8dfc 00007fa63d9f8236
0114 08:59:45.177413 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1887065856}
I20250114 08:59:45.179426 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3bfdda69d4934970f1e752db88d50" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e"
I20250114 08:59:45.180083 21547 raft_consensus.cc:3049] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Stepping down as leader of term 6
I20250114 08:59:45.180514 21547 raft_consensus.cc:738] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 28a83b7a7d9d4aa899b425086aa45a8e, State: Running, Role: LEADER
I20250114 08:59:45.135185 24604 raft_consensus.cc:513] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.131989 24761 consensus_queue.cc:1046] T b3650a91ff4a4213b9091f197bf75fb9 P d7732324fb5f48d3b16a29d4bb2584ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57f9cc1dd3c64647848a33a65b08c893" member_type: VOTER last_known_addr { host: "127.19.231.132" port: 35045 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250114 08:59:45.181414 21547 consensus_queue.cc:260] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.192749 21547 raft_consensus.cc:3054] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 6 FOLLOWER]: Advancing to term 7
W20250114 08:59:45.182533 21594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.129:45539, user_credentials={real_user=slave}} blocked reactor thread for 347922us
W20250114 08:59:45.189388 24656 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.135s user 0.016s sys 0.032s
I20250114 08:59:45.200609 21552 raft_consensus.cc:1235] T baef410f99d34fe9a53080abafb2259e P 28a83b7a7d9d4aa899b425086aa45a8e [term 10 FOLLOWER]: Rejecting Update request from peer cec0ef3bbb654988bf2123d8eef7a0a7 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20250114 08:59:45.206218 21547 raft_consensus.cc:2463] T fec3bfdda69d4934970f1e752db88d50 P 28a83b7a7d9d4aa899b425086aa45a8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0ef3bbb654988bf2123d8eef7a0a7 in term 7.
I20250114 08:59:45.207585 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2d1d6b0f14582b7864c1b80a47f26" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.208519 21547 raft_consensus.cc:2463] T e9e2d1d6b0f14582b7864c1b80a47f26 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 8.
I20250114 08:59:45.212420 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.212651 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5c7ffbe79e4145825848346fd7652c" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" is_pre_election: true
W20250114 08:59:45.199980 24656 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.146s user 0.017s sys 0.032s
I20250114 08:59:45.162004 24760 consensus_queue.cc:1035] T ada199404e61479593fa1b64c574bcb8 P 57f9cc1dd3c64647848a33a65b08c893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20250114 08:59:45.195327 24673 raft_consensus.cc:3049] T d19fa6dec61d44618fee387c9b1f2994 P 28a83b7a7d9d4aa899b425086aa45a8e [term 8 LEADER]: Stepping down as leader of term 8
I20250114 08:59:45.200706 24655 raft_consensus.cc:491] T abedb4afaad04af9bbbb7d4266fd2632 P d7732324fb5f48d3b16a29d4bb2584ed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 08:59:45.220983 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82151d61cef04050b69f24bc614507fc" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.237705 21594 leader_election.cc:304] T e9e2d1d6b0f14582b7864c1b80a47f26 P d7732324fb5f48d3b16a29d4bb2584ed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a83b7a7d9d4aa899b425086aa45a8e, d7732324fb5f48d3b16a29d4bb2584ed; no voters:
I20250114 08:59:45.209447 24184 consensus_queue.cc:1046] T 6a451ff14593400ca08caa1b5ed169f2 P 28a83b7a7d9d4aa899b425086aa45a8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20250114 08:59:45.221081 24366 leader_election.cc:290] T 106dea9d52554552a1b3e20e7c82a3a8 P 28a83b7a7d9d4aa899b425086aa45a8e [CANDIDATE]: Term 10 election: Requested vote from peers d7732324fb5f48d3b16a29d4bb2584ed (127.19.231.130:36891), cec0ef3bbb654988bf2123d8eef7a0a7 (127.19.231.131:37485)
I20250114 08:59:45.106712 24728 raft_consensus.cc:513] T 61f37f8d5b9d4c71b6e852f77fa438e8 P d7732324fb5f48d3b16a29d4bb2584ed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.085950 24702 raft_consensus.cc:513] T fec3bfdda69d4934970f1e752db88d50 P d7732324fb5f48d3b16a29d4bb2584ed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" member_type: VOTER last_known_addr { host: "127.19.231.129" port: 45539 } } peers { permanent_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" member_type: VOTER last_known_addr { host: "127.19.231.130" port: 36891 } } peers { permanent_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" member_type: VOTER last_known_addr { host: "127.19.231.131" port: 37485 } }
I20250114 08:59:45.243409 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba4257b18434159a50a4f6ae4a0d736" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:45.620790 22241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 397251us
I20250114 08:59:45.949745 21547 raft_consensus.cc:2463] T 3ba4257b18434159a50a4f6ae4a0d736 P 28a83b7a7d9d4aa899b425086aa45a8e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7732324fb5f48d3b16a29d4bb2584ed in term 11.
I20250114 08:59:45.396374 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa5b3173173431d91e459f9fdcb61a9" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.950961 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baef410f99d34fe9a53080abafb2259e" candidate_uuid: "d7732324fb5f48d3b16a29d4bb2584ed" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
W20250114 08:59:45.328013 21521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.231.131:37485, user_credentials={real_user=slave}} blocked reactor thread for 133775us
I20250114 08:59:45.952052 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f466e3efd6cb41329852e9028490bdf7" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "28a83b7a7d9d4aa899b425086aa45a8e" is_pre_election: true
I20250114 08:59:45.980680 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02341e5e574d55aac7eb9b3df9dc73" candidate_uuid: "cec0ef3bbb654988bf2123d8eef7a0a7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "